The Joys Of Programming

Radical Ans wrote:
trueheart78 wrote:

My C++ code has officially brought tears to someone's eyes?

I'll be putting that on my resume! ;)

Depends on what kind of tears. :)

Always the bad kind. C++ doesn't produce happy tears. Except when you don't have to look back...;)

I just picked up 7 languages/Databases in 7 weeks and am plowing through them now as I am so tired of my current code environment sapping all my skills. Time to stop letting this stupid platform atrophied my intelligence!

Dr.Ghastly wrote:

Time to stop letting this stupid platform atrophied my intelligence!

I'm not sure if I should laugh or cry at that...

trueheart78 wrote:
Dr.Ghastly wrote:

Time to stop letting this stupid platform atrophied my intelligence!

I'm not sure if I should laugh or cry at that... :lol:

Feel free to laugh at my crying.

Ok, awesome dudes I'm having a real java problem.

We have a java server that does some data analysis stuff it can spawn multiple threads to run concurrent jobs. Nothing too out of the ordinary there but we've just noticed that concurrent, and nominally isolated threads appear to be writing over each others UUID info. We're using the UUID for each thread to uniquely id some temp files that each (supposedly) isolated process generates.

we have a bit of code that does essentially this at the top of each child process

UUID rootUUID = java.util.UUID.randomUUID(); String tempNameRoot = "/tmp/" + rootUUID.toString(); DoThing1(UUID,tempNameRoot); DoThing2(UUID,tempNameRoot);

In testing when we spawn 2 processes one immediately after the other. The first child process sets it's UUID and runs DoThing1(). While the first process is running DoThing1() the second child process starts and calls java.util.UUID.randomUUID(). This also appears to set the first process's rootUUID to the same as the second process, causing the first process to completely fail.

That's the simplest distillation of the problem and we've tried a lot and have come up blank. Any ideas? We are completely stumped here.

We had thought that perhaps calls to java.util.UUID.randomUUID() are somehow global to the jvm so we tried this in the initialisation steps

UUID tmp_rootUUID = java.util.UUID.randomUUID(); String rootUUID = tmp_rootUUID.toString(); String tempNameRoot = "/tmp/" + rootUUID.toString();

But it has the same issue. tempNameRoot remains correct throughout

Hmm. May need more context to know the scope of these variables.

You mention multiple processes. These are actual processes, and not threads in a single process space? If so, and you're forking but not execing, it could be that they're setting up UUID state including the PRNG state equally first, and then both using that same state to generate the UUID, which would be bad. Not sure how to avoid that, since you don't get access to the underlying PRNG, unless you generate the UUIDs in the parent process instead of the child processes. You'll still have problems in the child processes if they ever want to generate UUIDs, if this is what's going on.

Hypatian wrote:

Hmm. May need more context to know the scope of these variables.

Sadly it would be hard to give much more context this is a pretty huge server. This is all happening in a org.apache.xmlrpc.server.XmlRpcServer;

Hypatian wrote:

You mention multiple processes. These are actual processes, and not threads in a single process space? If so, and you're forking but not execing, it could be that they're setting up UUID state including the PRNG state equally first, and then both using that same state to generate the UUID, which would be bad. Not sure how to avoid that, since you don't get access to the underlying PRNG, unless you generate the UUIDs in the parent process instead of the child processes. You'll still have problems in the child processes if they ever want to generate UUIDs, if this is what's going on.

What actually happens is that the parent thread instantiates a new instance of our job object, which is when the UUID call is initially made. The instance is actually a runnable object so is submitted to a queue for running. The DoThings() actions are actually in a run() method.

At this point we are right at the very edge of my java knowledge.

So these are definitely in threads, and not processes? If that's the case, I'm not sure how the value would be getting duplicated unless the same variable is somehow in scope in both of the threads, allowing the second thread that runs to clobber the first one. If you can have the worker threads print out their UUID for debugging as soon as they set it, you could see if they're getting different UUIDs but the second one is clobbering the first, or if they're getting the same UUID from two separate calls.

Is there a reason your aren't using File.createTempFile(prefix, suffix)? I don't use Java, and haven't used the equivalents in other languages much, but based on the docs, it looks like it handles the file name randomization for you. Your threads may still be clobbering each other, but at the very least it would give you more information.

Java concurrency can yield some oddness if your not careful. Defining a variable in on thread might not be seen by another thread and if you are sharing the same variable you could end up with weird results. One suggestion I can make is try adding a lock to the uuid variable to make sure that any changes to the variable are seen by the threads properly. It might slow things down but it might also give you some insight on the problem.

Typically when I am doing threading and I have a bunch of tasks to run, I prefer to setup the producer/consumer pattern so threads never access common variables and instead of fed tasks from a queue.

Could you alter the PRNG seed for the second process?

Hypatian wrote:

So these are definitely in threads, and not processes? If that's the case, I'm not sure how the value would be getting duplicated unless the same variable is somehow in scope in both of the threads, allowing the second thread that runs to clobber the first one. If you can have the worker threads print out their UUID for debugging as soon as they set it, you could see if they're getting different UUIDs but the second one is clobbering the first, or if they're getting the same UUID from two separate calls.

print out their UUID and the .timestamp()

have each print out their UUID and timestamp after the Dothing1 call, before tdothing2, as well.

If these are truly separate threads with locally initialized UUID's, there's no way that thread2 and change the value of thread1's rootUUID. That could only happen if rootUUID was a static variable on the thread class.

Edit, you could also dump a big chunk of your thread class to pastebin and link it here.

Hypatian wrote:

So these are definitely in threads, and not processes?

Don't know.

Hypatian wrote:

If you can have the worker threads print out their UUID for debugging as soon as they set it, you could see if they're getting different UUIDs but the second one is clobbering the first, or if they're getting the same UUID from two separate calls.

This we have done when thread 1 starts it processit sets UUID rootUUID it then starts DoThing1() before thread 2 starts. Then thread 2 stars sets a new UUID which clobbers thread 1's UUID so when process 1 exits DoThing1() when we check thread 1's UUID it's changed to match thread 2's.

Garden Ninja wrote:

Is there a reason your aren't using File.createTempFile(prefix, suffix)?

We want a given process to name all it's tmp files in a consistent manner because we need to do things with them and occasionally track them for debugging purposes

kazar wrote:

One suggestion I can make is try adding a lock to the uuid variable to make sure that any changes to the variable are seen by the threads properly. It might slow things down but it might also give you some insight on the problem.

Good idea.

Mixolyde wrote:

print out their UUID and the .timestamp()

have each print out their UUID and timestamp after the Dothing1 call, before tdothing2, as well.

Done that. We know where and when the over write of the initial thread's UUID happens and it's definitely thread 2 "clobbering" a variable that thread 1 has previously set.

Mixolyde wrote:

If these are truly separate threads with locally initialized UUID's, there's no way that thread2 and change the value of thread1's rootUUID. That could only happen if rootUUID was a static variable on the thread class.

That I will have to re-check. I don't recall seeing that it was declared as static though.

Mixolyde wrote:

Edit, you could also dump a big chunk of your thread class to pastebin and link it here.

Can do but it's several classes deep where the issue occurs, I could throw it all up pastebin

EDIT: What I really don't understand is why our tempNameRoot variable appears process/thread safe:

String tempNameRoot = "/tmp/" + rootUUID.toString();

but the UUID is not, even when declared in the same scope and even if we stringify it 'locally'

UUID rootUUID = java.util.UUID.randomUUID();
or

UUID tmp_rootUUID = java.util.UUID.randomUUID(); String rootUUID = tmp_rootUUID.toString();

Sounds like you are defining using the same UUID variable in multiple threads which is why they are clobbering each other. Can you pass the UUID to the "doThing" methods instead and make sure you are using a local copy of the UUID variable instead of a member or class version? If you absolutely need it to be a member variable, then I recommend to make a new object for each thread so they don't even know of each others values. And remember, globals are evil.

kazar wrote:

Sounds like you are defining using the same UUID variable in multiple threads which is why they are clobbering each other.

Yes that is exactly what it looks like. Which is why I moved the initialising of the UUID from anything that might be a parent thread or class of the runnable object. Now it happens in the class that is runnable and it still gets clobbered and as I say the tempNameRoot initialised at the same time does not get over written.

Can you pass the UUID to the "doThing" methods instead and make sure you are using a local copy of the UUID variable instead of a member or class version?

This is what is actually happening, I just simplified the example code above.

Oh hmm. It sounds like you're using the same Runnable for both threads. I'm thinking that the rootUUID is an attribute of that Runnable object, so of course if you set it in on thread you'll of course also set it in the other--you're modifying the same single Runnable. If you want to have mutable member variables in your Runnable and be using it for more than one thread at a time, you're going to need to make a new Runnable for each thread. Alternatively, you could store the value in some other object that is created uniquely for each thread.

Alternatively, if it's instead a local variable in the run function, things will work fine. But if you're tracking it in the Runnable, well, you just can't do that unless you only use the Runnable in one thread at a time.

Java sometimes does make the same variable point to different things in different threads. Only if you mark it as volatile or put it in a synchronized block do you get a guarentee that it is shared amongst all threads.

But something you said Hypatian reminded me of a possible solution. If DanB were to put the UUID in a ThreadLocal object, then every thread would have its own UUID so it wouldn't be possible for them to clash. All you have to do is initialize the ThreadLocal with a new UUID and when you are done your work, reset it so the next execution gets a new UUID.

If it's a private variable in the Thread, and you have

ThreadClass thread1 = new ThreadClass();
ThreadClass thread2 = new ThreadClass();

thread1.run();
thread2.run();

Shouldn't that prevent them from modifying each other's variables? You're not calling run on the same thread object multiple times, are you? Or, as Hypatian said, re-using the same Runnable object?

You could also look at the source for java.util.UUID.randomUUID(); and make sure that it's not doing anything funky. I don't have my environment for that up right now, or I'd look for you.

Solution found! So it turns out that one of my colleagues lectures our undergrads on Java Concurrency so we stepped through the code this morning

Coming back to this:

UUID rootUUID = java.util.UUID.randomUUID(); String tempNameRoot = "/tmp/" + rootUUID.toString(); DoThing1(UUID,tempNameRoot); DoThing2(UUID,tempNameRoot);

Apologies , I could probably have illuminated this problem much better in this example. In my rush to illustrate what is a beast of a piece of code I didn't give the best summary example. what really happens is something more akin to this:

UUID rootUUID = java.util.UUID.randomUUID(); String tempNameRoot = "/tmp/" + rootUUID.toString(); thing1 = new Thing1((UUID,tempNameRoot); thing2 = new Thing2((UUID,tempNameRoot); ... //many more thingX classes thing1.do(); thing2.do(); ... //many more thingX.do();

When ThingX instances are initialised it turned out that in some but not all of those classes the internal copy of UUID was set to static.

DanB wrote:

When ThingX instances are initialised it turned out that in some but not all of those classes the internal copy of UUID was set to static.

That will do it. "static" is one of those keywords that you want to be really careful with and only use it when you know you want to use it.

Mixolyde wrote:
DanB wrote:

When ThingX instances are initialised it turned out that in some but not all of those classes the internal copy of UUID was set to static.

That will do it. "static" is one of those keywords that you want to be really careful with and only use it when you know you want to use it.

It's kind of amazing that this bug has only just turned up after the server being live for the best part of 4 years now. And when I say "only just turned up" I mean "only just had a user complain about a specific problem that has shown us that this is happening"

On the upside I've learnt a lot about org.apache.xmlrpc.server.XmlRpcServer in the last 24hrs that I previously didn't know about

Rich Text, Poor Me
Our e-form designer is rather crude in terms of features. A few years ago we added RTF capabilities in the hopes of allowing a lot more control over layout, justification, etc. But the problem was that we send jobs to printers in basically 1 of 2 formats: GDI (big images) or PCL.

Our PCL(5) generation is hand-written and I could not figure out how to get PCL from the RTF content that printed out exactly as it should. I dissected some MS Word print jobs and saw that they embedded font characters directly in the document so that the printer would know exactly what the correct metrics were. The problem is that Windows TrueType fonts and the pre-installed printer fonts do not entirely line up. Unfortunately, embedding font glyphs is no easy matter and I haven't been able to find a good resource and exactly how to do it.

Hacks are bad?
So a total hack then: render the RTF content as an image and embed that image in the PCL stream. Horrid performance for large RTF blocks though. Slow to render and slow to print because of the size of the print job. Come to find out also that over long periods of time, this method lead to memory issues and performance degradation. Implemented thread pooling to try and avoid and issues of memory leaks as objects were created and disposed. This fixed memory issues; it didn't fix the performance problems over time. 30 second jobs ended up taking over 5 minutes if the print service ran continuously for a couple of hours.

Time and a Word
At this point I gave up on that tactic and began to flail about for alternatives. I remembered having seen some MS Word interop libraries in .NET. I'd once pondered the idea of printing native Word documents using it. Instead, I tried to see if I could programmatically paste the RTF content into an in-memory Word document and get Word to generate PCL. I could! It took some finagling... when printing to a file, the output format depends on the kind of driver you have set for your default printer. Also, a little brief non-interactive dialog pops up each time it prints to file.

I was able to generate a PCL file for each block of RTF content, then plug the meat of that PCL file into the entire page's PCL stream. But now came the big test: letting the print service run with hundreds of jobs across several asynchronous threads.

FAIL! .NET was throwing COM errors after only a handful of successful prints. Nothing I did helped any. F*cking COM!!!!! sh*tty and unreliable.

An Interesting Discovery
However, one very useful bit of information came out of this failed Word interop experiment. The PCL generated by this method did not have embeded font glyph info. Wait, so how did it turn out looking so pretty? Looking through the PCL (not raw, I have an app that makes it human readable), I discovered that Word was not sending whole lines of text to the printer. It was breaking up lines into words, groups of characters, or even individual characters, and each of these would have their own X,Y coords. Word seemed to know which characters had mismatched widths in the printer's installed fonts.

I didn't have that info, but what I could do would be to specify coords for each individual character. Exact coords for each letter was something I was able to get from the RTF content in .NET. So maybe the print job will be a bit larger than what MS Word or Word interop would generate, but far far smaller than rendering the RTF content as an image and embedding that in the PCL stream. And no worries about COM or default printers.

So right now is my load testing. So far it's been running for an hour with no increase in print job generation times, but the memory footprint is huge. That'll be my next step, figuring out why the memory is so high.

CLR Profiler is telling me that the .NET ContextLayoutManager is holding on to the rich text boxes I create to format the rich text and give me metrics on location. The hidden danger of creating a UI element in a thread (WPF controls have no Dispose() method, sadly). But apparently it's an easy fix to get the ContextLayoutManager to release them, so I'm trying that now.

Success. 500+ print jobs and the memory footprint stayed between 63 and 69 MB the entire time.

Quintin_Stone wrote:

Success. 500+ print jobs and the memory footprint stayed between 63 and 69 MB the entire time.

I have enjoyed this dev diary as the last time I did anything with printers it was writing postscript. Happy times :-/

This story is the reason I count my blessings that all of our company's printables aren't managed by IT or print nicely, like crystal reports. (Instead I get to wrestle with Crystal, which, really, ain't all that bad)

Ok, y'all. I have a problem. I have to learn asp.net and javascript in a week. I am in a senior project for my undergrad degree and I have a background in Java, some C/C#, and no web experience that would be worth mentioning. Any recommendations on how to approach this? Our client won't start expecting anything for a week or two, but I have to move on it fast. I thought about just going and getting a manual, but I am usually more of a hands on/audible learner. Any suggestions (including "just suck it up and get a reference book you moron") are welcome.

IlleBellus wrote:

Any suggestions (including "just suck it up and get a reference book you moron") are welcome.

Not sure if there is better advice

1) Start with the w3schools.com html, css, and javascript tutorials. Be current and do the HTML5 and CSS2 tutorials
2) Make a simple webpage, do some simple stuff with the javascript and css like moving page elements around, verify the contents of a web form
3) Now do the JQuery tutorial at w3schools, now switch to using JQuery to do all your javascript, it's cleaner to write and read and it handles all the cross browser stuff that you don't want to.
Maybe this: http://www.authenticsociety.com/blog...

IlleBellus wrote:

Ok, y'all. I have a problem. I have to learn asp.net and javascript in a week. I am in a senior project for my undergrad degree and I have a background in Java, some C/C#, and no web experience that would be worth mentioning. Any recommendations on how to approach this? Our client won't start expecting anything for a week or two, but I have to move on it fast. I thought about just going and getting a manual, but I am usually more of a hands on/audible learner. Any suggestions (including "just suck it up and get a reference book you moron") are welcome.

Pluralsight.com Stick with webforms over mvc if you have the option, it's easier to get started with.

bandit0013 wrote:
IlleBellus wrote:

Ok, y'all. I have a problem. I have to learn asp.net and javascript in a week. I am in a senior project for my undergrad degree and I have a background in Java, some C/C#, and no web experience that would be worth mentioning. Any recommendations on how to approach this? Our client won't start expecting anything for a week or two, but I have to move on it fast. I thought about just going and getting a manual, but I am usually more of a hands on/audible learner. Any suggestions (including "just suck it up and get a reference book you moron") are welcome.

Pluralsight.com Stick with webforms over mvc if you have the option, it's easier to get started with.

WebForms is awful and outdated. Go with MVC, even if it's a bit more difficult to grasp, which I don't really think it is.