Just as a reminder:
"So, malloc on Linux only fails if there isn’t enough memory for its control structures. It does not fail if there isn’t enough memory to fulfill the request." - http://scvalex.net/posts/6/
Malloc can also fail if you're out of address space without necessarily being out of memory.
NT does a much better job of separating these concepts than Unix-family operating systems do. Conceptually, setting aside a region of your process's address space and guaranteeing that the OS will be able to serve you a given number of pages are completely different operations. I wish more programs would use MAP_NORESERVE when they want the former without the latter. (I'm looking at you, Java.)
One day, perhaps when I am old and frail, we will achieve sanity and turn overcommit off by default. But we're a long way from being able to do that now.
These days, I describe malloc() as "a function which allocates address space", to avoid confusion. Which means it makes sense that malloc() returns NULL if you are out of address space, even if you have lots of memory. (But so many people don't check malloc()'s return anyway...)
This is a mostly-untrue statement because it makes unreliable assumptions about the host system. It depends on the vm.overcommit_memory setting and the programmer should never make assumptions about why or when malloc might fail. Read more on Rich Felker's excellent blog post here: http://ewontfix.com/3/
Another is if you are allocating lots of memory with alternating mprotect() permissions. On some systems (AIX for example) this uses up all of the memory for control structures WAY before hitting the address space limit (I've seen it fail after just a couple of GB).
An identifier must start with $, _, or any character in the Unicode categories “Uppercase letter (Lu)”, “Lowercase letter (Ll)”, “Titlecase letter (Lt)”, “Modifier letter (Lm)”, “Other letter (Lo)”, or “Letter number (Nl)”.
Is that actually a good thing? If I'm using \d to validate numbers (for example to check before string to int conversion, or IP address, phone number, or any other use), other unicode digits are not helpful to me.
It's great to support unicode, but I don't think the \d should have been extended this way. Add a \ud or something.
(Incidentally, this may explain the finding from http://stackoverflow.com/a/16622773/172322, as to why adding the RegexOptions.ECMAScript flag in the C# code eliminates the performance gap)
Also, when using Python 3.2 it seems to be the default behavior
Python 3.2.3 (default, Oct 19 2012, 20:10:41)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> re.match(r'\d', '੧')
<_sre.SRE_Match object at 0x7f188f6d4850>
\d A digit: [0-9]
\p{Digit} A decimal digit: [0-9]
which is actually somewhat depressing. I'd expect the named class to include the full Unicode digit set. It's surprising to see:
ab1234567890cd matched 1234567890
ab𝟣𝟤𝟥𝟦𝟧𝟨𝟩𝟪𝟫𝟢cd no match
from code using
Pattern.compile("(\\p{Digit}+)");
EDIT: and perhaps more surprising to see in the logs:
Exception in thread "main" java.lang.NumberFormatException: For input string: "𝟤𝟥𝟦𝟧"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:449)
Happens in PHP only if you enable Unicode regex handling via the /u modifier and are running libpcre 8.10 or later (which corresponds to PHP 5.3.4 and later, assuming you're using the bundled libpcre): http://3v4l.org/QD3k0
If you're using pcre directly from C code, this is controlled by specifying the PCRE_UCP flag to pcre_compile(). By default, \d and friends only match ASCII characters even if the PCRE_UTF8 flag is set.
I would be reluctant to rely on this until the Go documentation is clearer on the intended behavior. Right now it's very poorly specified. The regex doc[1] talks about "same general syntax" as Perl, but points to [2], which doesn't seem to understand what it's saying, describing '\d' in terms of its "Perl" meaning, but then saying that it's [0-9].
As a Perl developer that's been making the switch to Go, I've been caught out a few times with Go's no-so-Perl-like regular expression syntax. In fact I wish I knew about your 2nd link before now, because that could have saved me a few hours over recent months.
I'm not particularly fond of go, but "correctly handling unicode" can be subjective and case-dependent... I think making only minimal guarantees and punting to the application is often the only sane course.
> "correctly handling unicode" can be subjective and case-dependent...
So is correctly handling integers.
> I think making only minimal guarantees and punting to the application is often the only sane course.
That is completely and utterly crazy, the average developer has neither the knowledge nor the resources to make anything but a mess out of it without proper tools and APIs. Even with these (including a complete implementation of the unicode standard and its technical reports) unicode is already complex enough to deal with.
Of course it's not "completely and utterly crazy."
Not every app needs to deal with the enormous complexities implied by "full unicode support", and given the huge cost of that, there's a real place for a minimalist approach. If all I do with unicode is input strings from the user, store them in a database, and then later spit them out, I don't need to be able to do Turkish case-conversion, and I may not want to pay the cost of making it possible.
Certainly tools and APIs help for those cases where an app needs to do the sort of complicated text-processing that warrants "full" unicode support, but it's not at all clear that the proper place for such support is in the base language libraries. It's quite reasonable for the language implementors to say "if you want to do X, we'll support that, but if you want to do Y and Z, please use external library L."
> Not every app needs to deal with the enormous complexities implied by "full unicode support", and given the huge cost of that, there's a real place for a minimalist approach.
Not sure what point you're trying to make, I never said all applications had to make full use of all possible Unicode APIs, I said the language must expose them. Because if it doesn't, those who should use them will never become aware of them let alone use them.
> If all I do with unicode is input strings from the user, store them in a database, and then later spit them out, I don't need to be able to do Turkish case-conversion, and I may not want to pay the cost of making it possible.
So?
> It's quite reasonable for the language implementors to say "if you want to add numbers, we'll support that, but if you want to subtract or divide them, please use external library L."
Really?
Then again, considering Go's embedded contempt for non-US locales (see: datetime patterns) I'm not even sure why we're having this discussion, and since it's obvious they don't care for a non-US world it make sense that they wouldn't care for processing text.
And at the end of the day, you agree that Go has no provision for unicode handling, you just think it's all fine and dandy.
I'd argue that perl gets it right--as the default behavior, this behavior would gravely violate the principle of least surprise, but for the 0.01% of people who want \d to match ੧, there's no harm to making it available as an option you need to specifically request.
Your are right. I mixed up dynamic symbolic execution with invariant detection. I know there is something called DySy which does do invariant detection using dynamic symbolic execution.
