ADD_PROJECTILE is messing with new lines (prevents $ regexp)

[K4thos]

Whenever I'm analyzing IDS files I first trim them and fix newlines via this code:

Code: [Select]

REPLACE_TEXTUALLY ~%WNL%~ ~%LNL%~
REPLACE_TEXTUALLY ~[%TAB% ]+~ ~ ~
REPLACE_TEXTUALLY ~^[%TAB% ]+~ ~~
REPLACE_TEXTUALLY ~[%TAB% ]+$~ ~~btw. a new command called PRETY_PRINT_IDS would be useful here.

When using ADD_PROJECTILE I've noticed that above solution is not enough - each time you use ADD_PROJECTILE "$" regexp special character stops working, so you can no longer use for example FILE_CONTAINS_EVALUATED. In order to fix it you must once again make a COPY of the file and do REPLACE_TEXTUALLY ~%WNL%~ ~%LNL%~ before calling FILE_CONTAINS_EVALUATED or COUNT_REGEXP_INSTANCES that checks end of the line. Is this intended behaviour?

[Wisp]

Essentially, ADD_PROJECTILE converts the file into CRLF as a side-effect and $ apparently only matches against LF. I don't know if this has always been the case, or if it's something e.g., OCaml has changed at some point (what, exactly, is matched by $ is not entirely specified). It's not intended behaviour, but I am not yet certain how to resolve this.

[Wisp]

OCaml's position is that this is not a bug. $ only matches against LF because you are not supposed to use CRLF in code (but WeiDU does; bad WeiDU!). I don't see any entirely backward-compatible solutions to this. More thought required, perhaps.

[Mike1072]

Maybe we could add WeiDU commands to infer and modify the type of newlines used in a file.

Whenever an ADD_PROJECTILE is performed, the behind-the-scenes steps for appending to the .IDS could be:

Code: [Select]

1. infer newline type
2. convert to WNL
3. append line
4. convert back to original newline type

[Wisp]

Newline type isn't really inferable, since the same file may contain all kinds of newlines in no necessary order. This is especially likely for files which have been the target of appends or other edits. The most straightforward approach would probably be to just convert all newlines into the right kind at read-time and leave it at that.