Welcome to the Invelos forums. Please read the forum rules before posting.

Read access to our public forums is open to everyone. To post messages, a free registration is required.

If you have an Invelos account, sign in to post.

    Invelos Forums->DVD Profiler: Plugins Page: 1 2 3 ...52  Previous   Next
New Plugin: BulkEdit - Bulk Editing, XML and CSV Import of Profiles
Author Message
DVD Profiler Desktop and Mobile Registrantmediadogg
Aim high. Ride the wind.
Registered: March 18, 2007
Reputation: Highest Rating
United States Posts: 6,396
Posted:
PM this userVisit this user's homepageEdit postDirect link to this postReply with quote
(Please do not quote this post or significant portions thereof - it will be edited frequently).

This plugin allows for batch or "bulk" style editing of a subset of profile data fields, in a worksheet datagrid format. There are also XML and CSV Import features.

  • XML and CSV import are intended as a convenience feature to allow quick input of personal infomation associated with a DVD profile that is new to your database. Full profile information is obtained by doing a "refresh" from the official Invleos database.


  • The XML import does not modify existing profiles - just adds new ones. Also, there is no attempt to map or compensate for arbitrary formats. The Invelos XML export format, as well as the CollectorZ export format are supported.


  • The CSV import does not modify existing profiles - just adds new ones. Also, there is no attempt to map or compensate for arbitrary formats. Two of The plugin CSV Export formats, as well as the DVDSpot export format are supported.


  • Overview and Method Of Operation

    This plugin was designed to edit a relatively small number of profiles (say up to 200) at a time. It was originally conceived as a way to conveniently edit Location, Slot and Notes entries to support the LoadDVD plugin, but has been extended.

    To operate, first flag the desired profiles using any techniques you want, then call BulkEdit. Or you can edit profiles from your WishList, or Manual Profiles. There are functions for creating and managing Manual Profiles, adding / editing discs and validating changes before committing them to the database. The command is located under Tools\BulkEdit.

    I have attempted to implement procedures that minimize the opportunity for database corruption, however it is absolutely imperative that you backup your database / filters / layouts / html sections, etc. before using this plugin. I reinforce this by a mandatory response to a reminder each time you use it.

    So far, with all my testing, I have observed limited database damage caused by a bug, and only rarely, any impact on the DVD Profiler program, which I then fixed ASAP. In all cases, Profiler recovered after restart without needing a database restore (just make sure you always answer "No" if Profiler asks to delete an unrecognized record. Then exit and restart). At the time of release, V0.84 Beta (pending), I am not aware of any bugs that will either crash DVD Profiler or damage a database, and will vigorously pursue any such reports.

    The latest release is available at the BulkEdit new home.

    Latest Release adds the following items to the list of legacy functions:
    - Menu: DiscIDs > Validate Flagged Profiles DIsc IDs
    - Menu: DiscIDs > Add by Legacy DiscID
    - Feature: "Inserted DVD not recognized" popup is intercepted and replaced with a dialog that allows fixing the DiscID

    The Legacy plugin, along with substantial documentation is available here:

    List of Major features and functions
    - Load collections of profile data in a spreadsheet like format
    - Load by category, flagged, XML, CSV
    - Edit existing profile data (except cast and crew), or create new profiles from XML and CSV (people have recovered entire databases this way. Especially before there was restore from Online)
    - Automatic creation of unique manual profiles when profile ID is missing in imported data
    - Automatic Locality guessing when Locality is missing in imported data
    - Import specific non-profiler formats: CollectorZ, DVD Spot, (maybe others I forgot)
    - Customizable CSV import. Built-in automatic recognition of legacy DVDP "2.4" format and the CSV Export plugin formats
    - Select which columns of data get displayed
    - Rearrange column headings either in the selection list or output display
    - Sort entire sheet by clicking column header
    - Ability to force a text field that includes numbers to sort correctly by prefix padding
    - Drop downs created for all multi-value fields, even if not available in the API. BE crawls through the database and builds data source arrays from the set of values in the database. Less of that is need now, due to DVDP enhancements, so I am not sure how much "crawling" is still needed. For example, there used to be no way for a plugin to get a list of all the users or retailers internally. Now there is.
    - Simple Find / Replace within a column
    - Regex find / replace within a column
    - Row by row, prefix or append text to the contents of a text field
    - Row by row, prefix or append the contents of one column to another
    - VB / C# script execution. Results can update data in place, or be placed in work columns
    - Row by row, column copy (e.g. Copy Original Title into Sort Title)
    - Change simulation to test the impact of changes before making actual database updates
    - Change and error log to evaluate why some change didn't work or allow for manual undo
    - Save Validated changes makes real-time updates to the database from the spreadsheet
    - Export XML
    - Bulk load "personal data" - stuff that does not download from invelos, and update an exiting database, using the profile ID as a key. You could do mass updates of notes, Location, Slot, Links etc. that way.
    - "ScrapeGoat" feature. Using the forum handle of a user, access the user's public online collection and selectively add profiles to your own collection. Not personal data of course. Just the profile IDs are scraped from the web page, and then you can selectively download profiles from the Invelos Online.
    - QuickSearch feature (Alt-F9) searches the entire database - every field in a cached XML for a specific string. Ability to include / exclude cast / crew. Ability to refresh the XML cache before running the search. Profiles where the string is found are flagged. In addition to all fields, Collection Number, Purchase place and Purchase Price can be selected for some reason. Maybe in the past you couldn't filter for them. I don't remember.
    - Real time database update
    - Create manual, Disc ID, Child, Custom profile IDs (remember, Ken has added to the main program since BE was originally created. I haven't tried to systematically take stuff out.)
    - Adds several functions under Collection
      - Flag In-between
      - Flag Box Set
      - Flag Gifts
      - Flag Watched
      - Flagged Add Disc
      - Flagged Clone to Manual
      - Flagged Clone to Disc ID
      - Flagged Clone Children from DiscIDs
      - Flagged Delete All Discs
      - Flagged Clear Collection#
      - Flagged Flag Child Profiles
      - Flagged Flag Parents

    Oh, BIG CAVEAT: BulkEdit has always explicitly and intentionally avoided attempting to edit CAST and CREW. Import of cast and crew is supported, but all editing functions are relegated to much smarter people! 
    ===================================================================================
                        Notice: Considerations for using BulkEdit on DVDP Client of a remote database

    Generally, it is NOT recommended to make bulk changes from a remote database client. However, just as the main program, BulkEdit will adjust its capabilities in this mode in order to minimize problems.

    - A small delay has been added between bulk operations that save to the database
    - Some functions may not be available, or might work differently. For example, Bulk Cloning operations, such as "Clone To Manual," will not reflect changes in the client collection display. You must manually execute "Refresh Collection F5" in order to see the changes on the client. However, the changes will be reflected immediately on the server screen.

    For Bulk Operations, especially in spreadsheet mode, it is recommended to run BulkEdit either on the server, or on the client with a local copy of the database, which can then be backed up and restored to the server for future use. That being said, BulkEdit has been adjusted to work correctly for limited updates. It will be slower and subject to errors in the case of a poor network connection. 

    ===================================================================================

    Screen Shots

  • Main Screen

  • File Menu

  • Multi-Copy

  • XML Import

  • Export To Excel

  • Drop Down Editing Example(Studio)

  • Context Menu

  • Manual Profile Creation


  • Selected Usage Information:

  • Bulk loading a set of profiles.

  • CollectorZ Migration

  • DVDSpot Migration


  • Issues:

    - Performance improvements are in the works. Don't try to edit more than a few fields within 100 - 200 profiles. It wasn't designed for more than that. Profile crawler to pick up studios, etc. runs at about 10 profiles / sec. So, the average 2500 profile database will take 4 or 5 minutes. It is worth it - you only have to do it once, and afterwards, you can edit Studios and Place of Purchase.
    - Excel export is one-way.
    - Looking for a way to "edit users" from the context menu.
    - Can only update Loan Due if DVD currently on loan
    - Cannot edit Events
    - Cannot edit cast / crew , and there is absolutely no intention to: an excellent plugin exists for this already

    Integrity Strategy

    - The plugin does no "deletes" of profiles or discs, only "add" or modification. If you want to delete something, do it with the main program.
    - Simulation mode uses exactly the same code path as profile update, allowing updates to be aborted if errors are discovered
    - An error in any data field during validation will prevent the profile from being updated. It is skipped.
    - All data fields that have a limited set of legal values, are edited using drop-downs, similar in operation to the main program.
    - A log tracks changes to each cell of the datagrid
    - All manual profiles created with this plugin are tagged for easy identification
    - Each time a profile is edited, an "Edited" tag is created with date and timestamp
    Thanks for your support.
    Free Plugins available here.
    Advanced plugins available here.
    Hey, new product!!! BDPFrog.
     Last edited: by mediadogg
    DVD Profiler Desktop and Mobile Registrantmediadogg
    Aim high. Ride the wind.
    Registered: March 18, 2007
    Reputation: Highest Rating
    United States Posts: 6,396
    Posted:
    PM this userVisit this user's homepageEdit postDirect link to this postReply with quote
    Miscellaneous Editing Features
    - Pending changes are highligted: both the cell and the row.
    - Columns can be rearranged.
    - Profile crawler results persist over sessions and re-installation of the plugin.
    - The text box for Notes editing has been enhanced with a subset of editing controls, but you can also quickly pull up the DVD Profiler native editor with a right-click context menu.

    Enhancements(V0.84)
    - Option to suppress TAG timestamps duriing updates, or use Date only, instead of Date/Time
    - Group copy: "MultiCopy", including Find/Replace
    - XML validated import
    - Ability to clear the log (save already avialble)
    - LastEdited profile field updated after each edit
    Enhancements(V0.85)
    - Option to select displayed profile when Profile ID cell is clicked
    Enhancements(V0.86)
    - Added Locks to XML Export / Import
    - Enhanced handling of Purchase Places
    - Improved Excel/XML Export and XML Import
    Enhancements(V0.88)
    - CollectorZ XML import
    - various bug fixes
    Enhancements(V0.91)
    - "Clone To Manual" and "Clone To "Disc ID" added to Collection\Flagged
    - Context menu invoke of Cast/Crew Editor
    - Added "Exit - Clear Sheet Data and Log" to the File Menu
    - Fixed bug that flagged Wishlist when loaded
    - Added clarifying text to Multi-Copy panel
    Enhancements(V1.0)
    - Misc bug fixes
    - Locality guessing improved
        - CSV Import of three formats
        - plugin CSV Export 2.4 format
        - plugin CSV Export Column format
        - DVDSPot export format
    - Imported fields no longer restricted to personal
    Bug Fixes(V1.01)
    Enhancements(V1.02)
    - Ability to edit Locality
    Enhancements Under Consideration (Priority Sequence)
    - Support for new release of Profiler is ongoing and high priority
    - "Edit Users" on context menu, seamless round-trip
    - Edit Events (not delete - just add / change)

    Abandoned Enhancements
    -  Excel macro to create BulkEdit-compatible XML (no longer needed as of V0.86 improvements)
    Thanks for your support.
    Free Plugins available here.
    Advanced plugins available here.
    Hey, new product!!! BDPFrog.
     Last edited: by mediadogg
    DVD Profiler Desktop and Mobile Registrantmediadogg
    Aim high. Ride the wind.
    Registered: March 18, 2007
    Reputation: Highest Rating
    United States Posts: 6,396
    Posted:
    PM this userVisit this user's homepageEdit postDirect link to this postReply with quote
    Usage Notes
    (1) You can't update disc information if there are no discs. It is a borderline bug that I give no error messages when you try. Changes to the "null" disc appear to work, but nothing really gets put  into the database.
    (2) Until further notice, even though columns may be moved, the following sets of column sequences must be preserved after movement:
    - casetype, slipcover (Not bulk editable in Profiler 3.1 - use Edit Profile)
    - purchase price, purchase price currency
    - SRP, SRP currency
    - studio1, studio2, studio3
    - genre1, genre2, genre3
    (3) V0.83 adds "Edit Profile" to the context menu to circumvent sluggish double click.
    (4) V0.83 adds immediate selective update of the datagrid after context menu editing.
    (5) "Loan Due" is not set properly for technical reasons. I had erroneously thought that this was working. Should be fixed in Profiler 3.5 timeframe, if not before.
    (6) In all cases, if locality information is missing, then it will be generated by using the Rating if available, or by using the GS1 country code correcsponding to the UPC. If the UPC is missing, a manual profile ID will be generated. If a disc ID is specified as the profile ID, then locality will be generated from Rating if available. Locality defaults to "0" (US) in all cases.

    Open Bug Reports
    (1) Occasionally plugin will lose the datagrid. DVD Profiler does not crash, but must be restarted before the plugin works correctly again. Of course I will fix this as soon as I can reliably reproduce it and figure out what the problem is.
    (2) Notes editing will not accept certain characters such as ".", "!", "&" and "#". I don't know why yet.
    (3) Disc information fields such as Location may be incorrectly marked "Read Only" if the first profile retrieved has no discs. Workaround is to add a disc to the first profile, then the whole column will become editable.
    (4) It is possible for the CSV Import to incorrectly parse fields under certain conditions.

    Bugs Fixed
    (1) 08/01/2008: Add Manual Profile with Disc ID and Locality = US appends ".0" to ProfileID. Doesn't cause any errors, but DVD Pro does not append the ".0". (Fixed: V082)
    (2) 08/01/2008: Exception on Add Disc context menu. (Fixed: V082)
    (3) Add disc does not select the "Discs" panel. (Fixed V083)
    (4) Crawler does not distiguish between embedded values (e.g. 20th Century  incorrectly = 20th Century Fox ..) (Fixed V083. Must re-run Profile Crawler).
    (5) 08/17/08: Notes was sometimes not updated correctly during Column to Column copy. (Fixed V0.85)
    (6) WishList Priority "ReallyWantIt" is not set properly in edit mode. Fixed in V088.
    (7) Loan Due / Loan To not processed correctly during XML Import. Improved in V088.
    (8) Loading of Wishlist would flag profiles has been corrected (V091).
    (9) Copy of Genres during Profile Clone incorect - fixed. (V092)
    Thanks for your support.
    Free Plugins available here.
    Advanced plugins available here.
    Hey, new product!!! BDPFrog.
     Last edited: by mediadogg
    DVD Profiler Desktop and Mobile RegistrantStar ContributorNewEnglander
    Registered: 11/13/2003
    Registered: March 13, 2007
    Reputation: High Rating
    United States Posts: 1,911
    Posted:
    PM this userEmail this userView this user's DVD collectionEdit postDirect link to this postReply with quote
    Oh my gawd, woohoo, must try, must try thank you in advance a new toy!

    Would like to have seen a bulk edit of purchase place, as I needed a quick way to combine Walmart and Wal-Mart.
    Signature banned: Reason out of date...
     Last edited: by NewEnglander
    DVD Profiler Desktop and Mobile RegistrantStar ContributorNadja
    Small and broken
    Registered: March 13, 2007
    United Kingdom Posts: 775
    Posted:
    PM this userView this user's DVD collectionEdit postDirect link to this postReply with quote
    Jawesome!
    DVD Profiler Unlimited RegistrantStar Contributordee1959jay
    Registered: March 19, 2007
    Reputation: Highest Rating
    Netherlands Posts: 6,014
    Posted:
    PM this userView this user's DVD collectionEdit postDirect link to this postReply with quote
    Many thanks!! 
    DVD Profiler Desktop and Mobile Registrantmediadogg
    Aim high. Ride the wind.
    Registered: March 18, 2007
    Reputation: Highest Rating
    United States Posts: 6,396
    Posted:
    PM this userVisit this user's homepageEdit postDirect link to this postReply with quote
    Quoting NewEnglander:
    Quote:
    Oh my gawd, woohoo, must try, must try thank you in advance a new toy!

    Would like to have seen a bulk edit of purchase place, as I needed a quick way to combine Walmart and Wal-Mart.
    Ok, try this: (1) Use Database Query to flag the profiles with just those, (2) If you haven't already, run Profile Crawler (takes 10 profiles / sec), (3) Load Flagged after checking the Purchase Place box, and (4) make the changes. I recommend running the simulation first and taking a look at the log before committing the changes.

    If this thing looks semi-promising, I will do a "group paste", i.e. copy  "Wal-mart" into a bunch of selected cells at once. Now you have to do them one-by-one.

    Edit: As of V0.84, the change can be done simply by Find/Replace of "Wal-Mart" to "Walmart" and then SaveValidatedChanges. Couple of minutes and all done.
    Thanks for your support.
    Free Plugins available here.
    Advanced plugins available here.
    Hey, new product!!! BDPFrog.
     Last edited: by mediadogg
    DVD Profiler Desktop and Mobile RegistrantStar ContributorNewEnglander
    Registered: 11/13/2003
    Registered: March 13, 2007
    Reputation: High Rating
    United States Posts: 1,911
    Posted:
    PM this userEmail this userView this user's DVD collectionEdit postDirect link to this postReply with quote
    Quoting mediadogg:
    Quote:
    Quoting NewEnglander:
    Quote:
    Oh my gawd, woohoo, must try, must try thank you in advance a new toy!

    Would like to have seen a bulk edit of purchase place, as I needed a quick way to combine Walmart and Wal-Mart.
    Ok, try this: (1) Use Database Query to flag the profiles with just those, (2) If you haven't already, run Profile Crawler (takes 10 profiles / sec), (3) Load Flagged after checking the Purchase Place box, and (4) make the changes. I recommend running the simulation first and taking a look at the log before commiting the changes.

    If this thing looks semi-promising, I will do a "group paste", i.e. copy  "Wal-mart" into a bunch of selected cells at once. Now you have to do them one-by-one.



    Nice.

    May I ask why it takes additional steps to edit the purchase place than it does to change say currency type? Doesn't seem to make much sense to me. But, I am not a programer.
    Signature banned: Reason out of date...
    DVD Profiler Desktop and Mobile Registrantmediadogg
    Aim high. Ride the wind.
    Registered: March 18, 2007
    Reputation: Highest Rating
    United States Posts: 6,396
    Posted:
    PM this userVisit this user's homepageEdit postDirect link to this postReply with quote
    Quoting NewEnglander:
    Quote:
    May I ask why it takes additional steps to edit the purchase place than it does to change say currency type? Doesn't seem to make much sense to me. But, I am not a programer.
    Good question. Let me reiterate that you do the "Crawler" only once, per database.

    The reason is that fields such as Currency, Country, Ratings, Genre have specific values defined by DVD Profiler, and are available in a list for me to use. Other fields, such as users, purchase places, tags,  and studios are user-definable. There is no standard list. You can add, change or delete entries from those fields. Without going into the details - the only way I can figure out what values you use in those fields, is to"crawl" through the profiles and look at them, and make lists that I save to use subsequently. Each time you select a DVD, I take a quick peek to see if there is anything new.

    Other plugin developers understand the issue, so again, if they have any tricks up their sleeves that I can benefit from, I hope they let me know.

    Edit: And there are a couple of more types of fields, such as True/False - easy, and the text feilds where you are allowed to type in anything: Location, Slot, Label Side A, ... Notes, Overview, etc. They are also easy.
    Thanks for your support.
    Free Plugins available here.
    Advanced plugins available here.
    Hey, new product!!! BDPFrog.
     Last edited: by mediadogg
    DVD Profiler Desktop and Mobile RegistrantStar ContributorNewEnglander
    Registered: 11/13/2003
    Registered: March 13, 2007
    Reputation: High Rating
    United States Posts: 1,911
    Posted:
    PM this userEmail this userView this user's DVD collectionEdit postDirect link to this postReply with quote
    Quoting mediadogg:
    Quote:
    Quoting NewEnglander:
    Quote:
    May I ask why it takes additional steps to edit the purchase place than it does to change say currency type? Doesn't seem to make much sense to me. But, I am not a programer.
    Good question. Let me reiterate that you do the "Crawler" only once, per database.

    The reason is that fields such as Currency, Country, Ratings, Genre have specific values defined by DVD Profiler, and are available in a list for me to use. Other fields, such as users, purchase places, tags,  and studios are user-definable. There is no standard list. You can add, change or delete entries from those fields. Without going into the details - the only way I can figure out what values you use in those fields, is to"crawl" through the profiles and look at them, and make lists that I save to use subsequently. Each time you select a DVD, I take a quick peek to see if there is anything new.

    Other plugin developers understand the issue, so again, if you have any tricks up your sleeve that I can benefit from, let me know.

    Edit: And there are a couple of more types of fields, such as True/False - easy, and the text feilds where you are allowed to type in anything: Location, Slot, Label Side A, ... Notes, Overview, etc. They are also easy.


    Thanks for explaining it, makes sense now.
    Signature banned: Reason out of date...
    DVD Profiler Unlimited RegistrantStar Contributorruineddaydreams
    Registered: Dec. 2, 2002
    Registered: March 14, 2007
    United States Posts: 1,338
    Posted:
    PM this userEmail this userView this user's DVD collectionEdit postDirect link to this postReply with quote
    looks good, i will be trying this out very soon
    -JoN
    DVD Profiler Desktop and Mobile Registrantmediadogg
    Aim high. Ride the wind.
    Registered: March 18, 2007
    Reputation: Highest Rating
    United States Posts: 6,396
    Posted:
    PM this userVisit this user's homepageEdit postDirect link to this postReply with quote
    Version 0.82 available. See bugs fixed list at top of thread.
    Thanks for your support.
    Free Plugins available here.
    Advanced plugins available here.
    Hey, new product!!! BDPFrog.
    DVD Profiler Desktop and Mobile Registrantmediadogg
    Aim high. Ride the wind.
    Registered: March 18, 2007
    Reputation: Highest Rating
    United States Posts: 6,396
    Posted:
    PM this userVisit this user's homepageEdit postDirect link to this postReply with quote
    Version 0.83 available. See bugs fixed list at top of thread. Will incorporate support for the new release of Profiler as soon as possible. Anybody tried this thing yet? Results?
    Thanks for your support.
    Free Plugins available here.
    Advanced plugins available here.
    Hey, new product!!! BDPFrog.
     Last edited: by mediadogg
    DVD Profiler Unlimited Registrantjfrench
    Registered: April 16, 2008
    United States Posts: 347
    Posted:
    PM this userEdit postDirect link to this postReply with quote
    Hi,

    Great Addition!

    I was holding off in replying until I'd had a chance to get used to it.  It's already saved me a tremendous amount of time.

    I recently placed my DVDs in binders and as things progressed I saw the necessity of renaming each binder (I had given a descriptive name to each but then sorting was a mess). I put the names in the "Location" field and was confronted with the daunting task of editing every profile to reflect the changes.  Fortunately,  I had saved the contents of each binder as a flag set.  Using your program I was able to apply the changes to approximately 1700 DVDs in about 5% of the time it would have taken otherwise.

    Needless to say the proposal to allow batch changes would have cut the time to about 1%.  I probably have no need to expand on my views on that modification.

    When I saw the "Edit" tags I was a bit disturbed but then discovered how easy it was to remove them.
    My initial attempt to remove the tags resulted in an error message telling me which titles the tag was associated with.  As I had edited approximately 165 profiles in each group the the error message bottom was off the page and I couldn't clear it without restarting Profiler.  That was a Profiler problem not yours.

    I had some problems in clearing the "Sheet data" but discovered that by clearing any Profile fields which had been selected and then reloading the profile group the problem disappeared and I was able to clear it successfully.

    Another plus:  the listing showed some errors I had in the location of the second disc in a set and disclosed some manual profiles which didn't need to be.  The ability to edit the profile directly by double-clicking was great.

    Super!
    DVD Profiler Desktop and Mobile Registrantmediadogg
    Aim high. Ride the wind.
    Registered: March 18, 2007
    Reputation: Highest Rating
    United States Posts: 6,396
    Posted:
    PM this userVisit this user's homepageEdit postDirect link to this postReply with quote
    @jfrench

    I am dancing in the streets!    Thanks for that. Any suggestion on the TAGs is welcome. Maybe I should offer an option to turn them off, but I wanted to put as much stuff in as possible that would give people a comfort level, and an audit trail in case something goes wrong.
    Thanks for your support.
    Free Plugins available here.
    Advanced plugins available here.
    Hey, new product!!! BDPFrog.
    DVD Profiler Desktop and Mobile RegistrantStreetp
    A long time ago...
    Registered: March 19, 2007
    Canada Posts: 74
    Posted:
    PM this userView this user's DVD collectionEdit postDirect link to this postReply with quote
    Hi,

    This looks like it has the potential to be such a fantastic tool - when I first started collecting DVDs I put their location in the Notes field - since then improvements have been made to DVDProfiler so that there is a dedicated location and slot....

    This plugin will finally mean that I can delete all of those old Notes that are no longer valid. I'm just in the process of downloading, once I've tried it I'll let you know how it goes.

    Thanks
    To Err is human, to really louse things up takes a Computer......
        Invelos Forums->DVD Profiler: Plugins Page: 1 2 3 ...52  Previous   Next