marc_export utility allows the creation of invalid (too large) MARC records

I'm tempted to set this to "Won't fix," and add a comment along the lines of "MARC is a broken format. Don't use it."

However, I think this is more of an issue with MARC::Record and friends, since MARC::Record sets the size in the leader. I also think you should check what version of MARC::Record you have installed. I recall seeing code in a recent version that should handle oversize records by setting the size to 99999.

FWIW, I've only ever seen oversized records when exporting holdings, usually for the whole consortium. Most of our vendors have workarounds for this by ignoring the size field and reading to the next record separator. Really, any decent software should ignore the size field since it is wholly unnecessary.

You could try exporting records with holdings in separate batches for each member library. That should only take you and your vendor until Doomsday to output and to parse.

And, its practically 2015. Can we have a decent bibliographic record format already?

Oh, I should add:

1. I'm not opposed to adding a --debug option. Question is: what would it output?

2. I think if you want to reject oversize records that should also be added via a --strict option or some such. It might also benefit from changes to MARC::Record. It has been a few months since I last looked at the latter, so I don't remember off the top of my head if it can throw errors on oversized records.

> I also think you
> should check what version of MARC::Record you have installed. I recall
> seeing code in a recent version that should handle oversize records by
> setting the size to 99999.

I'm running 2.0.3, which is the packaged version for Ubuntu 12.04. 2.0.6 (the most recent version) is in the 14.04 repos.

> FWIW, I've only ever seen oversized records when exporting holdings,
> usually for the whole consortium.

Yes, that's what's triggering the issue. Since we have ~285 libraries adding holdings to the same bib records, this problem is more common with PINES than I would expect to see elsewhere.

> Most of our vendors have
> workarounds for this by ignoring the size field and reading to the next record
> separator. Really, any decent software should ignore the size field
> since it is wholly unnecessary.

Yes, at the moment, the vendor we're pushing our files to is working around the invalid records.

> 1. I'm not opposed to adding a --debug option. Question is: what
> would it output?

If possible, I think the bib record ID should be emitted along with any error/warning. That would be immensely helpful.

> 2. I think if you want to reject oversize records that should also be
> added via a --strict option or some such. It might also benefit from
> changes to MARC::Record. It has been a few months since I last looked
> at the latter, so I don't remember off the top of my head if it can
> throw errors on oversized records.

If it can, I like the idea of a --strict.

Thanks!

Chris

I am unsure of the status of this bug. I'm still tempted to set it to won't fix. I don't think there is much that we can do about oversized records in Evergreen, except possibly split the holdings up into separate, legally-sized records on output. This would mean breaking different rules by having multiple records in the same collection with the same value in 001.

In short, I don't think there is a good solution to the oversized record problem, except that everyone agree to ignore the size field. (Good luck with that!)

In the meantime bug 1502152 proposes a change to at least output the record id of the oversized records on export. While that bug/branch is targeted at master, it should backport to as far back as 2.7 without conflicts.