#antlr4

Screenshot I dusted off my toy compiler this weekend, and had a bit of fun implementing IF and ELSE support. The MLIR+LLVM compiler infrastructure is impressively versatile, allowing me to implement a feature like this in a handful of hours. It was, however, an invasive feature addition, requiring grammar changes, parser/builder, and lowering adjustments, as well as additional lowering for the…

Over the last 8 years, I’ve been intimately involved in building a pair of LLVM based compilers for the COBOL and PL/I languages.  However, a lot of my work was on the runtime side of the story. This was non-trivial work, with lots of complex interactions to figure out, but it also meant that I didn’t get to play with the fun (codegen) part of the compiler. Over the last month or so, I’ve been…

Тулза для работы с грамматиками показала себя очень хорошо.

Настолько, что у меня родился план: сделать парсер RenPy-грамматики не только на базе Python, но и на базе С#. Благо, antlr4 позволяет. Да что там парсер, [почти] полнофункциональный интерпретатор RenPy из Unity.

Renity!

Сказано — сделано. Пара бессонных ночей, свой форк antlr4-runtime — и скрипты RenPy парсятся из CLR, а корутины Unity выполняют базовые команды.

Что еще предстоит? Свой стак вызовов и работа с циклами и условиями. К чему приведет? Вероятно, к своему первому платному ассету.

Год назад [я сидел на скамейке в общественном парке], когда я выбирал, куда уходить с Fungus’а (визуального движка для VN’ок на Unity, не рекомендую), я пытался что-то прототипировать, но раскочегарить у меня не вышло: скилла не хватило. За год скилл вырос, пойти в интерпрайз на полгода было хорошей идеей.

Steps to configure InteliJ for using the ANTLR4 language with automated builds via Maven. This is as of 10 June, 2016 with IntelliJ IDEA Ultimate 2016.1 and Antlr 4.3. I’m using Ubuntu 16.04, but that probably doesn’t matter.

1) Download and install the ANTLR4 plugin. This will give you syntax highlighting, etc. You could also use this for building the grammer, but we’re going to use the Maven tool for that instead. Go to File | Settings... | Plugins | Browse Repositories | Antlr v4 Grammer Plugin to make the changes as shown below.

2) Set up a new maven project. In that project we’re going to add two folder hierarchies. The first is src/main/antlr4. This will hold the ANTLR grammers. You put each grammer in a subdirectory corresponding to the package you want the Java grammer files to be generated to. In this case, I’m creating a grammer called com.hablutzel.jutzo.parser.Jutzo, so the file hierarchy looks like the below.

Doing thing this way allows you to keep the ANTLR 4 sources separate from the Java sources, and also keeps them separate from the Java classes that ANTLR will generate. However, we still have to compile the ANTLR grammar. You can do that manually, when you have the grammar open (CTRL-SHIFT-G), but we want this to happen manually. We’ll do that with a Maven build plugin in the next step.

(3) Configure the pom.xml file to build the ANTLR grammar into Java files. This can be done with the antlr4-maven-plugin, but there is an undocumented feature that lets you change where the files get generated. Normally they go to the target/generated-sources folder, but InteliJ IDEA maven projects are generally set up to exclude the entire target directory, so you can’t add the generated-sources as a source folder. Instead, we’ll change the source folder using the <outputDirectory> configuration element so that it points to gen/main/antlr4 instead; the actual Java files generated will be in the right package structure under that directory (so the grammar Jutzo.g4 above will go to gen/main/antlr4/com/hablutzel/jutzo/parser/*)

<build>     <plugins>         <plugin>             <groupId>org.antlr</groupId>             <artifactId>antlr4-maven-plugin</artifactId>             <version>4.5.3</version>             <configuration>                 <outputDirectory>gen/main/antlr4</outputDirectory>             </configuration>             <executions>                 <execution>                     <id>antlr4</id>                     <goals>                         <goal>antlr4</goal>                     </goals>                 </execution>             </executions>         </plugin>     </plugins> </build>

The build plugin requires Maven 3, so you have to change that. In File | Settings | Maven make sure you’re using Maven 3:

(4) OK, now we have to add the gen/main/antlr4 as a source directory. Do this with File | Project Structure | Modules | Sources, as shown below.

(5) Almost there. We have the Maven target, but nothing to invoke it when we do a build. In order to do that, we have to add a maven goal task as the first step of the make. In Run | Edit Configurations, add “Run Maven Goal” a before launch task (the little green +). Given the above build plugin configuration, which has an execution ID of “antlr4″ and a goal of “antlr4″, the poorly named “command line”, which is really the maven goal, is “antlr4:antlr4″. (That’s executionid:goal).

Make sure that goal is the first before launch task:

(6) [Optional] if you want to make sure that the ctrl-shift-G generates the grammar in the same place, open the grammar, right-shift in the editor window, and select “Configure ANTLR...”. Change the output directory to the fully qualified path name under the gen/main/antlr4 directory - in this example, you have to use the entire gen/main/antlr4/com/hablutzel/jutzo/parser path. You only have to do this if you want to use ctrl-shift-G to generate the grammar.

This is useful if you want to build the project without running it, as I haven’t figured out how to make the build trigger the maven task. If you figure that out, please comment and I’ll add it.

(7) Finally you have to make sure that the project has ANTLR support at runtime. Back to the pom file, and add the dependency:

<dependencies>     <dependency>         <groupId>org.antlr</groupId>         <artifactId>antlr4</artifactId>         <version>4.5.3</version>     </dependency> </dependencies>

Testing grammar using ANTLR4 TestRig (Grun)

“Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.” –Martin Golding

Quotes like above tell us the importance of testing the modules that we develop. Testing for different languages is done using various tools for example JUNIT for java, FunSuite for scala and TestRig (grun on command line) for grammar(.g4). In this blog our focus…

ANTLR คืออะไร

อ่านว่า Antler ย่อมาจาก  ANother Tool for Language Recognition

ANTLR เป็นโปรแกรมสำหรับรับ input grammar file และ auto generate output source code for recognizer (recognize คล้ายๆกับ RegEx)

ทำไมต้อง ANTLR 

ใช้ RegEx ไม่ได้หรอ

ประโยชน์ที่รู้ตอนนี้คือ เราสามารถเขียน rule ง่ายๆ โปรแกรม ANTLR จะสามารถ gen code ให้เป็นภาษาตามที่เราต้องการได้ ไม่ว่าจะเป็น python, java, java script, c, c++, ruby

ประโยชน์อีกข้อคือ หากในอนาคตมีการเปลี่ยนแปลง rule ก็ไม่ต้องไปแก้โค้ดใหม่ ให้ ANTLR gen ใหม่ให้ เพียงกดแค่คลิกเดียว สบายมากๆ

ประโยชน์อื่นๆ ตาม document ที่เค้าว่า RegEx ทำไม่ได้ คือ

1. สามารถทำ complex parsing ได้ 2. support ‘error-messages’ ANTLR จะshow ด้วยว่า row ไหน column ไหน ที่ผิด ซึ่ง RegEx ทำไม่ได้

if you need error-messages, complicated things, ANTLR is good option

สรุป

ANTLR is code generator. It takes so called grammar file as input and generates two classes: lexer and parser.

Lexer runs first and splits input into pieces called tokens

Parser who builds abstract syntax tree, interprets the code or translate it into some other form

Grammar file contains everything ANTLR needs to generate correct lexer and parser

วิธีติดตั้ง ANTLR

ปัจจุบัน (2016) เป็น version 4.5.x แต่ไม่ support python แหะ 

prerequired

antlr-x.y-complete.jar  --> download the "Complete ANTLR x.y Java binaries jar" file

antlr python runtime --> http://www.antlr3.org/download/

Install ANTLR

1. download the “Complete ANTLR x.y Java binaries jar” file

$ wget http://www.antlr.org/download/antlr-3.5.2-complete.jar

2. Add ANTLR to your CLASSPATH  environmental variable and run it

สำหรับ v3

export CLASSPATH=~/antlr-3.5.2-complete.jar:$CLASSPATH java org.antlr.Tool -version

สำหรับ v4

export CLASSPATH=~/antlr-4.5.1-complete.jar:$CLASSPATH java org.antlr.v4.Tool 

3. Configure your CLASSPATH to include ANTLR on future logins

echo "export CLASSPATH=~/antlr-3.5.2-complete.jar:$CLASSPATH" >> ~/.bashrc

4. วิธีการติดตั้ง python runtime

ไปที่ folder python runtime พิมพ์คำสั่ง

$ python setup.py install

5.  สร้าง rule ( file .g )

http://www.needfulsoftware.com/ANTLR3/Python

6. เมือมี rule file .g แล้ว ขั้นตอนต่อไปคือ การ generate 2 files:

Grammar1Lexer.py

Grammar1Parser.py

ให้พิมพ์คำสั่ง:

สำหรับ v3

$ $ java -classpath "antlr-3.5.2-complete.jar" org.antlr.Tool ./Playspace/ANTLR/v3/Python/Grammar1/*.g

สำหรับ v4

$ java -classpath "antlr-4.5.1-complete.jar" org.antlr.v4.Tool ./Playspace/ANTLR/v4/MyGrammar/*.g

7. ขั้นตอนสุดท้าย ทดสอบ เรียกใช้ rule