Welcome to the Invelos forums. Please read the forum rules before posting.

Read access to our public forums is open to everyone. To post messages, a free registration is required.

If you have an Invelos account, sign in to post.

    Invelos Forums->General: Website Discussion Page: 1... 17 18 19 20 21 ...26  Previous   Next
goodguy's Credit Lookup Plus
Author Message
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,045
Posted:
PM this userVisit this user's homepageDirect link to this postReply with quote
Quoting GSyren:
Quote:
Quoting mediadogg:
Quote:
Aha, so even CLTPlus does not always 100% match CLT! I didn't get any parsing errors when loading the file, and it has only 2559 profiles, not 2560 as CLT reports. Anyway, still better than 2539. 

Hang on maybe spoke too soon ... doing a quick check, why I am missing one on load ...

You should go through all entries in the CLT manually, and see if one of them maybe has an error!   

Oh yeah, I tried that once. Feel asleep before I got finished. Started over. Repeated until realization of stupidity set in. Wrote a program. Fell asleep while testing. Started over ...   
Thanks for your support.
Free Plugins available here.
Advanced plugins available here.
Hey, new product!!! BDPFrog.
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,045
Posted:
PM this userVisit this user's homepageDirect link to this postReply with quote
Gonna take a break for a day or so. Something is going on with the web site? Don't want to waste time, so I will wait for a bit to make sure.
Thanks for your support.
Free Plugins available here.
Advanced plugins available here.
Hey, new product!!! BDPFrog.
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,045
Posted:
PM this userVisit this user's homepageDirect link to this postReply with quote
Darn. After completing more debugging options so I could track down why I miss a small percentage of profiles sometimes (unfortunately for actors with thousands of profiles), I was enjoying posting on my recipes thread while waiting for the test to finish. Finally, it finished, clicked on the results screen ... and CRASH!!! Lost it all. That usually means out of memory, because almost everything else is error trapped ...

Are the Gods of Invelos against me??? 
Thanks for your support.
Free Plugins available here.
Advanced plugins available here.
Hey, new product!!! BDPFrog.
 Last edited: by mediadogg
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,045
Posted:
PM this userVisit this user's homepageDirect link to this postReply with quote
Splitting out to separate plugin today. After bugs cleanup, maybe that will resolve some of the other issues, such as memory crashes. Yeah, I can hear you thinking ... should have done it long ago.   
Thanks for your support.
Free Plugins available here.
Advanced plugins available here.
Hey, new product!!! BDPFrog.
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,045
Posted:
PM this userVisit this user's homepageDirect link to this postReply with quote
Finally completed the extraction, and the resulting plugin does seem to be faster than when it was running as part of BulkEdit. Now back to testing, using the Danny Devito CLTPlus data as a comparison. I compute a running estimate of completion, which now says 1 hour 40 min. So, its off to bed, and I will see what's up in the morning.
Thanks for your support.
Free Plugins available here.
Advanced plugins available here.
Hey, new product!!! BDPFrog.
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,045
Posted:
PM this userVisit this user's homepageDirect link to this postReply with quote
Gunnar, after being reminded of CefSharp, I downloaded and took another look today. Dead fast, easy, yummy.

The difference is already apparent from simply right-clicking on show page source. After clicking on the credit URL in the embedded IE browser, you can't see the info.

In Chrome on the desktop, you get the dreaded "confirm res-submission".

But in CefSharp, you see the entire credit detail HTML division, ripe for the scraping. Be still my heart.

Unless, you are already in progress, or intend to be, I propose to drop CLTBoss, at least temporarily and see how fast I can cobble up a quick CLT scrape. No plugin needed.

I would happily turn over the unpolished code to you, just to save you the trouble of slogging through the scraping. You could then, if you would like to, include the scrape directly in your tool and "Goodbye GoodGuy".

Now all CLTBoss needs to do simply grab XML from the online, because of CefSharp can deliver the complete profile ID without the messy scrape do-do I have been in, then we are sailing on to glory.

So what do you say? I will be spending the day seeing how fast a scrape I can cobble up, by porting over my logic from CLTBoss. If I can get to the credit detail, it is game over.
Thanks for your support.
Free Plugins available here.
Advanced plugins available here.
Hey, new product!!! BDPFrog.
 Last edited: by mediadogg
DVD Profiler Unlimited RegistrantStar ContributorGSyren
Profiling since 2001
Registered: March 14, 2007
Reputation: Highest Rating
Sweden Posts: 4,049
Posted:
PM this userVisit this user's homepageView this user's DVD collectionDirect link to this postReply with quote
With just screenscraping you can't get all the data that CLTBoss was extracting, right? So this would pretty much just return what CLTPlus did, but not give us anything more? No full profiles?
My freeware tools for DVD Profiler users.
Gunnar
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,045
Posted:
PM this userVisit this user's homepageDirect link to this postReply with quote
Quoting GSyren:
Quote:
With just screenscraping you can't get all the data that CLTBoss was extracting, right? So this would pretty much just return what CLTPlus did, but not give us anything more? No full profiles?

Right, but here is what I was thinking:

(1) People would like to have a replacement for CLTPlus - that could be CLTInfo

(2) I would still work on CLTBoss, but now its scraping would be optional, and use the same CefSharp code (if I get it working).

The biggest, and highly useful difference is that CLTInfo would be a complete tool, with no dependencies that could replace and go beyond CLTPlus. As a big added plus, it could also load XML that has been downloaded from the online, and do even more glorious statistics.
Thanks for your support.
Free Plugins available here.
Advanced plugins available here.
Hey, new product!!! BDPFrog.
 Last edited: by mediadogg
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,045
Posted:
PM this userVisit this user's homepageDirect link to this postReply with quote
Of course this is all moot if I discover some unforeseen roadblock that still prevents me from completing a fast and accurate scrape.

Edit: Not making much progress. The net is full of people asking the same questions, and the answers given are the very same lines of code that don't work. Will switch back to CLTBoss testing if I don't get a breakthrough by tonight. The fundamental issue of getting no notice when the credit link is clicked is still the case. Breakpoints in all the handlers. Nada. So there will still be a need to set timers, take a guess, wait for it ..

The big difference is that at least if you wait, the data will actually be there. But I haven't found a single sample program that works other than to pop up notepad with the browser contents.
Thanks for your support.
Free Plugins available here.
Advanced plugins available here.
Hey, new product!!! BDPFrog.
 Last edited: by mediadogg
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,045
Posted:
PM this userVisit this user's homepageDirect link to this postReply with quote
Shaking in my boots. Figured it out. Afraid to hope beyond hope, but it looks promising. Still requires a small delay (500 ms), but I have been able to click on a page and see it change, then click on a credits link, wait 500 ms and then grab the browser contents for scraping. Simple  tests, but it was fast and reliable.

If you guys are still tolerating my old muddle-brained antics, I am going to take a few hours of think time, have dinner and wine with my wife, and then I will cobble together the most basic, fastest "quick-scrape" that I can come up with. Then it will be available to anybody who wants it, and I will also retrofit it into CLTBoss.
Thanks for your support.
Free Plugins available here.
Advanced plugins available here.
Hey, new product!!! BDPFrog.
 Last edited: by mediadogg
DVD Profiler Unlimited RegistrantStar ContributorGSyren
Profiling since 2001
Registered: March 14, 2007
Reputation: Highest Rating
Sweden Posts: 4,049
Posted:
PM this userVisit this user's homepageView this user's DVD collectionDirect link to this postReply with quote
Quoting mediadogg:
Quote:
I would happily turn over the unpolished code to you, just to save you the trouble of slogging through the scraping. You could then, if you would like to, include the scrape directly in your tool and "Goodbye GoodGuy".

I'm afraid I'll have to decline. Several reasons, most importantly:
- C# and VB doesn't mix, and I don't do C#. There may also be Forms / WPF incompatabilities.
- CLTInfo is done and just awaiting final testing against CLTBoss output, and I don't really want to rewrite it again.

I am also heavily into a new project that is more interesting for me personally.

If anyone wants to try to marry your screen scraping code with my CLTInfo code, I'll gladly hand over my code. Just know that it's WPF and doesn't lend itself to conversion to Windows Forms.
My freeware tools for DVD Profiler users.
Gunnar
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,045
Posted:
PM this userVisit this user's homepageDirect link to this postReply with quote
Thanks for the detailed reply. I will go ahead and put the results into CLTBoss, if I am successful.
Thanks for your support.
Free Plugins available here.
Advanced plugins available here.
Hey, new product!!! BDPFrog.
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,045
Posted:
PM this userVisit this user's homepageDirect link to this postReply with quote
Another Preview

This shows scraping for UPCs and subsequent discovery of associated profiles and then scanning the XML from those profiles to extract credits. The variants are actually two unrelated actors in this case.

I am still working on the problem of handling the XML output for an actor with thousands of profiles credited.

Also, I was not able to eliminate network issues. Even that fancy Chrome browser has timing issues, so I have put that back on the shelf for a future upgrade. It is fast, and I might be able to get around the issues once I learn it better.

CLTBoss has been split out from BulkEdit and is now a separate plugin. Once again, I apologize for the amount of time it is taking me. 

Edit: I just found the ZipFile class. Maybe thats the ticket!
Thanks for your support.
Free Plugins available here.
Advanced plugins available here.
Hey, new product!!! BDPFrog.
 Last edited: by mediadogg
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,045
Posted:
PM this userVisit this user's homepageDirect link to this postReply with quote
I think I got around the memory issue without using zip compression. Still testing.

Meanwhile I am retrofitting the XPath search to the minimalist CLTBoss XML, and I have gotten myself confused about group and episode dividers.

Correct me if I am wrong, but are the groups and episode designations not available in the XML, or am I missing something? I am trying to avoid losing the advantages of the Xpath by needing to manually loop through all the credits anyway, just to correctly identify groups and episodes.

Can someone help clear my foggy brain. I am so loopy from all the rewrites and testing, etc that I have forgotten some stuff I ought to know. 
Thanks for your support.
Free Plugins available here.
Advanced plugins available here.
Hey, new product!!! BDPFrog.
 Last edited: by mediadogg
DVD Profiler Unlimited RegistrantStar ContributorGSyren
Profiling since 2001
Registered: March 14, 2007
Reputation: Highest Rating
Sweden Posts: 4,049
Posted:
PM this userVisit this user's homepageView this user's DVD collectionDirect link to this postReply with quote
Looking for something like this?

    <Credits>
      <Divider Caption="CLTBoss and CLTInfo contributions" Type="Episode"/>
      <Divider Caption="The Good Old Boys" Type="Group" CreditType="Writing"/>
      <Credit FirstName="mediadogg" MiddleName="" LastName="" BirthYear="0" CreditType="Writing" CreditSubtype="Writer" CreditedAs=""/>
      <Credit FirstName="GSyren" MiddleName="" LastName="" BirthYear="0" CreditType="Writing" CreditSubtype="Writer" CreditedAs=""/>
      <Divider Caption="" Type="EndDiv" CreditType="Writing"/>
    </Credits>
My freeware tools for DVD Profiler users.
Gunnar
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,045
Posted:
PM this userVisit this user's homepageDirect link to this postReply with quote
Geez, I must be really losing it. Last night I looked at what I thought was an XML sample that should have contained those entries and got me all loopy. Today is another day. Thanks!

I've usually parsed divider credits only from the API, not the XML, but XPath works on the XML and when the guts of the credit are revealed to me in the XPath extension, dividers are already gone. Tricky.
Thanks for your support.
Free Plugins available here.
Advanced plugins available here.
Hey, new product!!! BDPFrog.
    Invelos Forums->General: Website Discussion Page: 1... 17 18 19 20 21 ...26  Previous   Next