It's not my favorite api but for users that currently use the same thing
from `parser`, moving over is more difficult
than it needs to be.
If you plan to support both old and new ruby versions, you definitly need to
branch somewhere on the ruby version
to either choose prism or parser.
But with prism you then need to enumerate all the versions again and choose the correct one.
Also, don't recommend to use `Prism::Translation::Parser` in docs. It's version-less
but actually always just uses Ruby 3.4 which is probably
not what the user intended.
Note: parser also warns when the patch version doesn't match what it expects. But I don't think prism has such a concept,
and anyways it would require releases anytime ruby releases, which I don't think is very desirable
https://github.com/ruby/prism/commit/77177f9e92
This changes reference_count on rb_method_definition_struct into an
atomic.
Ractors can create additional references as part of `bind_call` or
(presumably) similar. Because this can be done inside Ractors, we should
use a lock or atomics so that we don't race and avoid incrementing.
Co-authored-by: wanabe <s.wanabe@gmail.com>
Blocks insert a leading `nop` instruction in order to execute a "block
call" tracepoint. Block compilation unconditionally inserts a leading
`nop` plus a label after the instruction:
641f15b1c6/prism_compile.c (L6867-L6869)
This `nop` instruction is used entirely for firing the block entry
tracepoint. The label exists so that the block can contain a loop but
the block entry tracepoint is executed only once.
For example, the following code is an infinite loop, but should only
execute the b_call tracepoint once:
```ruby
-> { redo }.call
```
Previous to this commit, we would eliminate the `nop` instruction, but
only if there were no other jump instructions inside the block. This
means that the following code would still contain a leading `nop` even
though the label following the `nop` is unused:
```ruby
-> { nil if bar }
```
```
== disasm: #<ISeq:block in <main>@test.rb:1 (1,2)-(1,17)> (catch: FALSE)
0000 nop ( 1)[Bc]
0001 putself [Li]
0002 opt_send_without_block <calldata!mid:bar, argc:0, FCALL|VCALL|ARGS_SIMPLE>
0004 branchunless 8
0006 putnil
0007 leave [Br]
0008 putnil
0009 leave [Br]
```
This commit checks to see if the label inserted after the `nop` is
actually a jump target. If it's not a jump target, then we should be
safe to eliminate the leading `nop`:
```
> build-master/miniruby --dump=insns test.rb
== disasm: #<ISeq:<main>@test.rb:1 (1,0)-(1,17)>
0000 putspecialobject 1 ( 1)[Li]
0002 send <calldata!mid:lambda, argc:0, FCALL>, block in <main>
0005 leave
== disasm: #<ISeq:block in <main>@test.rb:1 (1,2)-(1,17)>
0000 putself ( 1)[LiBc]
0001 opt_send_without_block <calldata!mid:bar, argc:0, FCALL|VCALL|ARGS_SIMPLE>
0003 branchunless 7
0005 putnil
0006 leave [Br]
0007 putnil
0008 leave [Br]
```
We have a test for b_call tracepoints that use `redo` here:
aebf96f371/test/ruby/test_settracefunc.rb (L1728-L1736)
Instead of requiring the consumer to provide a list of all events which
they wish to handle, we can give them to option of dynamically detecting
them, by scanning the listener's public methods.
This approach is similar to that used by Minitest (scanning for `test_`
methods) and Rails generators (running all public methods in the order
they are defined).
While this is slower than specifying a hard coded list, the penalty is
only during registration. There is no change the the behaviour of
dispatching the events.
https://github.com/ruby/prism/commit/781ebed743
[Misc #21143]
[Bug #21193]
The previous change caused a backward compatibility issue with code
that called `Object.const_source_location` from the `inherited` callback.
To fix this, the order is now:
- Define the constant
- Invoke `inherited`
- Invoke `const_set`
Since `rb_bug` does not always take Ruby frame info during SEGV, the
source file path may not be output.
```
1) Failure:
TestRubyOptions#test_crash_report_script [/tmp/ruby/src/trunk_gcc11/test/ruby/test_rubyoptions.rb:907]:
Expected /
bug\.rb:(?:1:)?\s\[BUG\]\sSegmentation\sfault.*\n
/x
to match
"[BUG] Segmentation fault at 0x000003e900328766\n"+
```
http://ci.rvm.jp/results/trunk_gcc11@ruby-sp2-noble-docker/5663880
This can get triggered even if the list of statements only contains
a single statement. This is necessary to properly support compiling
```ruby
defined? (;a)
defined? (a;)
```
as "expression". Previously these were parsed as statements lists
with single statements in them.
https://github.com/ruby/prism/commit/b63b5d67a9
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
Of course, these won't really be fixtures, but it allows to test against whole codebases
without copying them, doing symlinks or something like that.
For example, I can tell that over the whole RuboCop codebase, there are only 8 files that produce mismatched ast.
Telling what the problem is is a different problem. The ast for real files can and will be huge so I haven't checked yet
(maybe parser bug) but it's nice for discoverability regardless
https://github.com/ruby/prism/commit/2184d82ba6
I see `Array.include?` as 2.4% runtime. Probably because of `LPAREN_CONVERSION_TOKEN_TYPES` but
the others will be faster as well.
Also remove some inline array checks. They are specifically optimized in Ruby since 3.4, but for now prism is for >= 2.7
https://github.com/ruby/prism/commit/ca9500a3fc
`Integer#chr` performs some validation that we don't want/need. Octal escapes can go above 255, where it will then raise trying to convert.
`append_as_bytes` actually allows to pass a number, so we can just skip that call.
Although, on older rubies of course we still need to handle this in the polyfill.
I don't really like using `pack` but don't know of another way to do so.
For the utf-8 escapes, this is not an issue. Invalid utf-8 in these is simply a syntax error.
https://github.com/ruby/prism/commit/161c606b1f
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
Turns out, it was already almost correct. If you disregard \c and \M style escapes, only a single character is allowed to be escaped in a regex so most tests passed already.
There was also a mistake where the wrong value was constructed for the ast, this is now fixed.
One test fails because of this, but I'm fairly sure it is because of a parser bug. For `/\“/`, the backslash is supposed to be removed because it is a multibyte character. But tbh,
I don't entirely understand all the rules.
Fixes more than half of the remaining ast differences for rubocop tests
https://github.com/ruby/prism/commit/e1c75f304b
The offset cache contains an entry for each byte so it can't be accessed via the string length.
Adds tests for all variants except for this:
```
"fo
o" "ba
’"
```
For some reason, this still has the wrong offset.
https://github.com/ruby/prism/commit/a651126458
There are a few other locations that should be included in that check.
I think the end location must always be present but I left it in to be safe (maybe implicit begin somehow?)
https://github.com/ruby/prism/commit/545d07ddc3
`Integer#chr` performs some validation that we don't want/need. Octal escapes can go above 255, where it will then raise trying to convert.
`append_as_bytes` actually allows to pass a number, so we can just skip that call.
Although, on older rubies of course we still need to handle this in the polyfill.
I don't really like using `pack` but don't know of another way to do so.
For the utf-8 escapes, this is not an issue. Invalid utf-8 in these is simply a syntax error.
https://github.com/ruby/prism/commit/161c606b1f
Mostly around newlines and line continuation.
* percent arrays need special backslash handling in the ast
* Fix offset issue for heredocs with many line continuations (used wrong variable as index access)
* More refined rules on when to simplify string tokens
* Handle line continuations in squiggly heredocs
* Correctly dedent squiggly heredocs with interpolation
* Consider `':foo:` and `%s[foo]` to not be interpolation
https://github.com/ruby/prism/commit/4edfe9d981
I want to add new node types to the parser translator, for example `itblock`. The bulk of the work is already done by prism itself. In the `parser`
builder, this would be a 5-line change at most but we don't control that here.
Instead, we can add our own builder and either overwrite the few methods we need,
or just inline the complete builder. I'm not sure yet which would be better.
`rubocop-ast` uses its own builder for `parser`. For this to correctly work, it must explicitly choose to extend the
prism builder and use it, same as it currently chooses to use a different parser when prism is used.
I'd like to enforce that the builder for prism extends its custom one since it will lead to
some pretty weird issues otherwise. But first, I'd like to change `rubocop-ast` to make use of this.
https://github.com/ruby/prism/commit/b080e608a8
1. The string starts out as binary
2. `ち` is appended, forcing it back into utf-8
3. Some invalid byte sequences are tried to append
> incompatible character encodings: UTF-8 and BINARY (ASCII-8BIT)
This makes use of my wish to use `append_as_bytes`. Unfortunatly that method is rather new
so it needs a fallback
https://github.com/ruby/prism/commit/e31e94a775
