1.. _contributions: 2 3====================================================================== 4Contributions 5====================================================================== 6 7:Maintainer: Masatake YAMATO <yamato@redhat.com> 8 9.. contents:: `Table of contents` 10 :depth: 3 11 :local: 12 13---- 14 15You are welcome. 16 17Supporting many parsers with few developers is impossible. We invite 18the person who contributes a parser to u-ctags team, especially if the 19target language is updated frequently. TypeScript is a typical 20frequently updated language. 21 22This is what we would like potential contributors to know. In this 23section "you" means a contributor, and "we" means reviewers. "I" means 24Masatake YAMATO, the author of this section. 25 26This page gathers random notes for newly joined members. 27 28Basic rules 29--------------------------------------------------------------------- 30 31You are the maintainer of your parser, of course. 32 33You may update your parser as you want under the rules described 34later. 35 36You may review pull requests changing your parser. 37 38A parser exists and is maintained independently form other 39parsers. However, considering the consistency between parsers are not 40bad. 41 42You can put your name to docs/developers.rst. 43 44Read docs.ctags.io. 45 46Before You Start 47--------------------------------------------------------------------- 48 49 .. Specific to add new parser and/or new kind/role 50 51When working on ctags I take into account the following uses for 52tags: 53 541. inserting the name with completion, 552. jumping to the definition of the name (in an editor or similar tool), 563. navigating the source code tree, 574. summarizing the source code tree, and 585. answering a query about the source code tree. 59 60When I review new parser code, I expect the parser to contribute to 61these purposes. 62 63What should be tagged? 64~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 65 66There are two classes of tags. The primary class is a *definition tag*. 67If a name is defined in a file, the name and the line and the file 68where the name is defined should be tagged (recorded). However, in 69some languages answering, "What is a definition?" is not so obvious. 70You may have to decide what is tagged in your parser thoughtfully. 71The purposes listed at the top of this subsection should help you 72decide. 73 74The secondary class is a *reference tag*. This is newly introduced in 75Universal Ctags and is not available in Exuberant Ctags. If a name is 76used (or referenced) in a file, it can be tagged as a reference tag. 77 78Don't be confused by the two tag classes. 79 80Defining kinds and roles 81~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 82 83Defining kinds is the most important task in writing a new parser. 84Once a kind is introduced, we cannot change it because it breaks 85tags file compatibility. 86 87See :ref:`tag_entries` in :ref:`ctags(1) <ctags(1)>` for more details of kinds 88and roles. 89 90Scope information and full qualified tags 91~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 92 93Optional. 94TBW. 95 96Developing a parser 97--------------------------------------------------------------------- 98 99Origin of changes and license 100~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 101 102Make clear where the patches come from and who wrote them. 103 104If you backport patches from Geany or some other project, their 105commit IDs should be logged, too. 106 107Include a copyright notice when adding a new 108``{parsers,main}/*.[ch]`` file. 109A new file also requires a license notice at the head of the file. 110 111We expect your change (or new code) to be provided under the terms of 112the General Public License version 2 or any later version. We would 113like you to express "version 2 or any later version". 114 115Reference 116~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 117 118In the comment at the head of your source file, include a URL for a 119web page that explains the language your parser deals with. 120Especially if the language is not well known. 121 122Here is an example. 123 124.. code-block:: C 125 126 /* 127 * 128 * Copyright (c) 2016, Masatake YAMATO 129 * Copyright (c) 2016, Red Hat, K.K. 130 * 131 * This source code is released for free distribution under the terms of the 132 * GNU General Public License version 2 or (at your option) any later version. 133 * 134 * This module contains functions for generating tags for property list defined 135 * in http://www.apple.com/DTDs/PropertyList-1.0.dtd. 136 */ 137 138C language 139~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 140 141Don't forget to use `static` modifiers. Don't introduce unnecessary 142global variables. 143 144Remove unused variables and types. If you want to keep them in your 145source code, include a descriptive comment. 146 147Use the available facilities provided by the ctags core. If the 148facilities are not enough for writing a parser, consider extending 149the core first. 150 151Use underscores in names only in file scope objects. 152Don't use them in function declarations, variable declarations or 153macro names in header files. 154 155Basic whitespace settings are specified in the `EditorConfig 156<https://editorconfig.org/>`_ configuration file (`.editorconfig`). 157There are `plugins <https://editorconfig.org/#download>`_ available 158for most popular editors to automatically configure these settings. 159 160Style guidelines are largely captured in the `Uncrustify 161<http://uncrustify.sourceforge.net/>`_ configuration file 162(`.uncrustify.cfg`). Formatting can be checked with: 163 164.. code-block:: console 165 166 $ uncrustify -c .uncrustify.cfg -f parsers/awk.c | diff -u parsers/awk.c - 167 168Don't mix `whitespace cleanup` fixes and other improvements in one 169commit when changing the existing code. Style fixes, including 170`whitespace cleanup`, should be in a separate commit. Mixing 171functional changes with style fixes makes reviewing harder. 172 173If possible, don't use file static variables. Find an alternative way 174that uses parameters. 175 176 177.. NOT REVIEWED YET 178 179Notes for GNU emacs users 180......................................................................... 181 182If you use GNU emacs, utilize the `.editorconfig` configuration based 183on non-GNU C style. Here non-GNU C style means 184"align a keyword for control flow and `{` of the block start". 185 186GNU style: 187 188.. code-block:: C 189 190 if (...) 191 { 192 ... 193 194non-GNU style: 195 196.. code-block:: C 197 198 if (...) 199 { 200 ... 201 202For combining the style and `.editorconfig` configuration, put 203following code snippet to your .emacs: 204 205.. code-block:: emacs 206 207 (add-hook 'hack-local-variables-hook 208 (lambda () (editorconfig-apply))) 209 210`.dir-locals.el` in ctags source tree applies "linux" style of `cc-mode`. 211Above code snippet applies the `.editorconfig` configuration AFTER 212installing the "linux" style to the current buffer. 213 214I like GNU style, but for keeping consistency in existing code of 215Exuberant Ctags, the origin of Universal Ctags, I introduced the style 216and configuration to my .emacs. Please, do the same. 217 218Compatibility 219~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 220 221We are trying to maintain compatibility with Exuberant-ctags in the 222following two areas. 223 224* Command line option 225* Tag file compatibility 226 227Command line options 228~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 229 230Don't introduce `--<LANG>-foo=...` style options. They are less 231suitable for command-line completion by the zsh/bash completion 232engines. Instead, introduce `--foo-<LANG>=...` style options. 233 234Add an entry to docs/news.rst if you change the behavior of an option 235or introduce a new option. If you think the option is stable enough, 236add it to ctags.1.in, too. 237 238Use underscore as a prefix for experimental options. Once an option 239is introduced, it must be maintained. We don't want to remove it 240later. If you are not sure of the usefulness of the option, use an 241underscore at the start of a long option name like: `--_echo`. 242 243Write a test case for Tmain or Units. 244 245Don't remove an option, especially if it exists in Exuberant Ctags. 246 247Writing parser 248~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 249There are two ways to write a parser, writing in C and using *optlib parser*. 250 251Universal Ctags extends the *optlib parser* feature so extensively that it can 252implement most of functions of a parser. 253*optlib parser* is also suitable for prototyping. 254 255See :ref:`ctags-optlib(7) <ctags-optlib(7)>` and :ref:`optlib` for details. 256See :ref:`optlib2c` how to add a optlib parser on ``ctags``. 257 258For writing a parser in C see :ref:`writing_parser_in_c`. 259 260Build script 261~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 262To add your optlib parser, ``foo.ctags``, into ``ctags`` do the following steps; 263 264* put ``foo.ctags`` file on ``optlib/`` directory 265* add ``foo.ctags`` on ``OPTLIB2C_INPUT`` variable in ``source.mak`` 266* add ``fooParser`` on ``PARSER_LIST`` macro variable in ``main/parser_p.h`` 267* add ``foo`` on the list in the section "New parsers" in ``docs/news.rst`` 268* add ``"..\optlib\foo.c"`` in ``win32/ctags_vs2013.vcxproj`` 269* add ``"..\optlib\foo.c"`` in ``win32/ctags_vs2013.vcxproj.filters`` 270 271Translated C code is also committed to our git repository. The translated code 272is useful for building ctags on the platforms where optlib2c doesn't run. 273 274To add your parser file, ``foo.c``, into ``ctags`` do the following steps; 275 276* put ``foo.c`` file on ``parsers/`` directory 277* add ``foo.c`` on ``PARSER_SRCS`` variable in ``sources.mak`` 278* add ``foo`` on the list in the section "New parsers" in ``docs/news.rst`` 279* add ``"..\parsers\foo.c"`` in ``win32/ctags_vs2013.vcxproj`` 280* add ``"..\parsers\foo.c"`` in ``win32/ctags_vs2013.vcxproj.filters`` 281 282If you have GNU make, ``make -C win32`` updates the win32 files described above 283from ``makefile.mak``. 284Without updating win32 files our CI process run on 285Appveyor will fail. 286 287See `this pull request <https://github.com/universal-ctags/ctags/pull/2765>`_ 288for the `Meson` parser as an example of optlib parser. 289 290Testing 291--------------------------------------------------------------------- 292 293Add test cases, and run both existing cases and your new cases. 294 295If you add a new parser or modify an existing parser, add new test 296cases to "Units". If you modify the core, add new test cases to 297"Tmain". The way to write and run test cases is described in 298:ref:`testing_ctags` and :ref:`testing_parser` section of this guide. 299 300With the exception of the tmain test harness, you can specify VG=1 301for running test cases under the Valgrind memory debugger. 302 303A parse should not enter an infinite loop for bad input. 304A parse should not crash for bad input. 305A parse should return control to its caller for bad input. 306 307Describe what kind of tests are passed in the commit message. 308e.g. :: 309 310 make units LANGUAGES=TTCN VG=1 is passed. 311 make fuzz LANGUAGES=TTCN VG=1 is passed. 312 make chop LANGUAGES=TTCN VG=1 is passed. 313 314Test cases 315~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 316 317Add a test case to Unit when creating or modifying a parser. 318 319Add a test case to Tmain when modifying the core. 320 321Add a test case to Tinst when modifying the install target in the 322Makefile. 323 324Testing your parser 325~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 326 327If possible, prepare a simple test and a complex one. The simple one 328for helping us, the maintainers, understand the intent of the 329modification. 330 331If there are more than 3 test cases for a parser, a parser specific 332test case directory should be prepared like `Units/parser-c.r`. 333 334Add realistic examples for you parser to *codebase* 335~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 336 337At `codebase <https://github.com/universal-ctags/codebase>`_, we 338collect realistic examples that can be used for evaluating your parser 339especially about its performance aspect. Consider contributing to the 340repository when adding a new parser to Universal Ctags. 341 342Writing Documents 343--------------------------------------------------------------------- 344 345* ``man/*.rst`` files are the source files of our man pages. 346 The man pages are for users. See "`How to add a new man page for your parser`_". 347 348* ``docs/*.rst`` files explain experimental 349 new features. The files are for developers. The parts of contents 350 of ``docs/*.rst`` should be moved to ``man/*.rst`` in the future. 351 352* Update ``docs/news.rst`` especially if you add a new parser. 353 354* Write ``docs/parser-<NAME-OF-YOUR-PARSER>.rst`` as you want. 355 A FAQ and the design or your parser are common topics. 356 Consider the maintenance of your parser after you left the 357 project for some reason. 358 359How to add a new man page for your parser 360~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 361 3621. write what the users of your parser may want to (or should) know to ``man/ctags-lang-LANGUAGE.7.rst.in`` 3632. add ``man/ctags-lang-LANGUAGE.7`` to ``GEN_IN_MAN_FILES`` of ``man/Makefile.am``. 3643. run ``make -C man update-docs``. This step generates the rst file at ``docs/man/ctags-lang-LANGUAGE.7.rst``. 3654. add ``ctags-lang-LANGUAGE(7)`` to (toctree of) ``docs/man-pages.rst``. 3665. do ``git add`` for 367 * ``man/ctags-lang-LANGUAGE.7.rst.in`` 368 * ``docs/man/ctags-lang-LANGUAGE.7.rst`` 3696. git commit with a log header: "``docs(man): add a man page for LANGUAGE``". 3707. make a pull request 371 372.. TODO: Why do we have to git add ``docs/man/ctags-lang-LANGUAGE.7.rst``? 373 A system which uses it has Sphinx. It should be able to generate 374 ``docs/man/ctags-lang-LANGUAGE.7.rst``. 375 376Committing and submitting a pull request 377--------------------------------------------------------------------- 378 379* Make a pull request even if the change is small enough. 380 381* Wait for one day till merging even if the change is small enough. 382 383* Wait for 3 days at least for non-small change to your parser. 384 385* Wait for 7 days at least and get an LGTM (Looks Good To Me) comment from a 386 member of the team if your commit changes the other parts than your parser and 387 the changes are not obvious. 388 389* Add a test case to your pull request. To make git-bisect happy, 390 don't add a test case for a feature or a bugfix before adding the 391 code for the feature or the bugfix. 392 393* Even if a pull request includes multiple commits, each commit must 394 be semantically well separated. Sometimes you may want to adjust 395 whitespaces in the code. Adjusting whitespaces is o.k., but don't 396 mix the other change with it. Make a commit just for the whitespaces 397 adjustment. 398 399Title of commit log and pull request 400~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 401 402* "Misc Fixes" is allowed as far as each commit in a pull request is 403 semantically well separated. Sometimes, you may fix various minor 404 things randomly. Making pull requests for each of them is 405 boring. You may want to make "mix fixes" pull request especially if 406 your code is young. 407 408* Use [WIP] (Work In Progress) prefix as the title of your pull request, if you don't 409 want people to take time for reviewing your code. Removing [WIP] 410 implies "ready to be reviewed." 411 412* Use [FYI] (For Your Information) prefix as the title to show your idea or sketch represented 413 in C language. 414 415* Use the name of your parser as the prefix of a commit log. 416 417 .. code-block:: git 418 419 C++: record template type parameters to detect the end of template prefix 420 421 If we know Foo is a name of type, it becomes easier to detect whether 422 ">>" in "Foo>>" is a shift operator or the end marker of the template 423 prefix. 424 425 In the above example, "C++: " is the prefix. 426 427* Use the name of your parser as the prefix of a pull request if your 428 change is about a parser. 429 430* Use following prefixes for the changes other than parsers. 431 432 main: 433 Changes for files under ``main/`` directory 434 435 Units: 436 Changes for the test cases under ``Units/`` directory 437 438 Tmain 439 Changes for the test cases under ``Tmain/`` directory 440 441 docs(web) 442 Changes for the ``docs/*.rst`` 443 444 docs(man) 445 Changes for the ``man/*.rst`` 446 447 See also the output of ``git log`` command. 448 449* Combine prefixes with a comma if a change modifies multiple parts of our source tree 450 451 Here is an example. 452 453 .. code-block:: git 454 455 456 commit 64a05963c108af4b7832a2215006ff5cafcaaebb 457 Author: Masatake YAMATO <yamato@redhat.com> 458 Date: Tue Mar 19 12:19:37 2019 +0900 459 460 main,Flex,JavaScript,SQL,refactor: introduce a helper function to skip two character sequence 461 462 ... 463 464* Use following prefixes if the change as no run-time impact. 465 466 cosmetic 467 - Remove whitespaces at the end of lines 468 - Adjust indentation 469 - Remove an empty line 470 - ... 471 472 style 473 - Rename symbol names 474 - ... 475 476 refactor 477 - Code transformation that doesn't intent changing run-time behavior 478 479 These prefixes reduce the load of reviewers. 480 481* Use [INCOMPATIBLE] as a prefix for both pull request and commit log 482 if the change breaks the compatibility with Exuberant Ctags. Write 483 an explanation in ``man/ctags-incompatibilities.7.rst.in`` about the 484 detail of breakage. 485 486* Use [SELF-INCOMPATIBLE] as a prefix for both pull request and commit 487 log if the change breaks the compatibility with Universal Ctags 488 itself. 489 490Commit log 491~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 492 493(For new parsers the following criteria is not applicable.) 494 495Make clear the original motivation for the change and/or the impact 496on the tags file. 497 498If you fix a bug reported somewhere on the web, its URL should be 499logged, too. 500 501If the bug is reported in the Exuberant Ctags tracker on the 502SourceForge web site, log it as ``sf-bugs:N``, ``sf-patches:N``, 503``sf-support-requests:N``, or ``sf-feature-requests:N``. 504``docs/tracking.rst`` also should be updated. 505 506Squashing commits 507~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 508 509When you submit a pull request you might receive some comments from a 510reviewer and, in response, update your patches. After updating, we 511would like you to squash your patches into logical units of work 512before we merge them to keep the repository history as simple as 513possible. 514 515* Use ``git rebase -i`` and ``git push --force`` to refine your change in 516 the meaning of "semantically well separated." "semantically well 517 separated" is important than "recording the history of your try and 518 error." 519 520Quoted from @steveno in `#393 521<https://github.com/universal-ctags/ctags/issues/393>`_ : 522 523 You can check out this page for a good example of how to squash 524 commits 525 http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html 526 527 Once you've squashed all your commits, simply do a git push -f to 528 your fork, and GitHub will update the pull request for you 529 automatically. 530 531Rules for reviewing a pull request 532--------------------------------------------------------------------- 533 534* Put your rough schedule as a comment if you don't have time, but you 535 want to review. 536 537Testing a PR locally before being merged 538~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 539 540You may want to test a PR locally before it is merged into the master 541repository. 542 543If you want to test a PR #2234 on a repository `upstream` as branch name 544`BRANCHNAME`;:: 545 546 $ git fetch upstream pull/2234/head:BRANCHNAME 547 $ git checkout BRANCHNAME 548 549This creates a branch `BRANCHNAME`, and pulls the PR into the branch, and 550switches to the branch. The branch name `BRANCHNAME` does not have to be the 551same as the branch name of the PR. 552 553Alternatively suppose you want to test ``USERNAME``'s PR with branch 554name ``main-fix-foo``;:: 555 556 git checkout -b tmp-main-fix-foo master 557 git pull https://github.com/USERNAME/ctags.git main-fix-foo 558 559This creates a branch ``tmp-main-fix-fix-foo`` from a branch ``master`` and 560switches to it, then pulls the branch ``main-fix-foo`` from 561``https://github.com/USERNAME/ctags.git``. 562