Open Source Integrated Library System

Evergreen on IRC

#openils-evergreen Logs for Monday, February 2nd, 2009

< Sunday, February 1st, 2009Raw Log FileTuesday, February 3rd, 2009 >
#TimeNickMessage
#02:11:00dbs has quit IRC
#03:15:48ort-9217 has quit IRC
#07:23:15natschil has joined #openils-evergreen
#07:24:56BenO has joined #openils-evergreen
#07:42:51natschil has quit IRC
#08:10:16kgs has joined #openils-evergreen
#10:23:13dbs has joined #openils-evergreen
#10:36:13kgs has quit IRC
#10:39:26kgs has joined #openils-evergreen
#11:04:40dbshmm, how do y'all feel about massaging the slim MODS so that multiple titleInfo/title elements can be returned?
#11:05:49dbsor at the very least, massaging the XPath so that the first titleInfo/title element is returned. use case is for bilingual MARC records that have two 245 fields, one English, one French
#11:06:00miker_dbs: I have an alternative for you .... :)
#11:06:05dbs(or one Nepali, one romanized transliteration)
#11:06:18dbsmiker_: you usually do :)
#11:06:23miker_;)
#11:07:47miker_so, we add a display_field flag on config.metabib_field
#11:07:50dbsbtw, that set of nepali records I added to the test datasets has one nice corrupt non-UTF8 field
#11:08:28miker_as part of ingest, we pull out those bits of the record, but we don't go normalizing them
#11:09:01miker_and we insert each instance into a separate row
#11:09:32miker_so, multiple 246 (translated title, I think?) fields go in separately
#11:10:05miker_then, instead of grabbing an MVR, we grab all the extracted fields from metabib.*_field_entry
#11:10:28miker_and display them as needed in the opac
#11:11:35miker_we'll need more on config.metabib_field (a label column, probably an ordinal position for the field row within the field_class)
#11:12:58miker_and, instead of "not normalizing" (or normalizing, in the case of search_field=t) there should be set of available normalizing transforms linked to each row in config.metabib_field (or a {something} separated list of such, in a text field)
#11:14:25miker_dbs: this is the idea that's been bouncing around in my head for a while ... I have not had two properly shaped tuits to rub together, though ...
#11:16:11granthpal has joined #openils-evergreen
#11:29:23miker_dbs: did I frighten you away?
#11:29:25miker_:)
#11:31:18miker_dbs: anyway, the end result would be instead of an mvr object, you'd get a HoH, like { title : { proper : ['x','y'], translated : ['z'] }, author : { personal : ['rowling'] } }
#11:31:27dbsmiker_: sorry, my director stopped by to chat
#11:32:21miker_dbs: btw, awesome slides. thanks for 1) doing that and 2) putting them up!
#11:32:45jeffsay, speaking of metabibs...
#11:32:56jeffmiker_: any way you know of to report on "bibs with no metabib"?
#11:32:59dbsahh, but just like in that nepali.marc example, there are two ways of providing transliterations; two 245 fields, or a 245 and a linked 880
#11:33:03jeff(other than at the database level)
#11:33:37dbsmiker_: well, more blog posts to come to include the code snippets and such for the other bits, but I've been meaning to pull that stuff together for a long time
#11:34:07dbsI do like the HoH approach though, for sure
#11:35:13_dkyle_ has joined #openils-evergreen
#11:44:24dbsI'm more worried, to be honest, about why only 20 of the 98 Nepali records ingest at all; looks like an OpenSRF JSON decoding issue
#11:48:10miker_jeff: you should be able to start from bib records, go to metarecord source map, and filter where source map id is null ... that works out to (something like) SELECT {blah} FROM biblio.record_entry b LEFT JOIN metabib.metarecord_source_map m ON (b.id = m.source) WHERE m.id IS NULL; ... which is the query you want
#11:49:36miker_dbs: even if we stick with mvr, I agree we should make it more easily extended ... store the extraction bits in the db, make everthing in the mvr an array, etc
#11:50:09miker_but I'd like to move to HoH, because of the flexiblity, and it seems simpler
#11:51:51dbsmiker_: right. by "set of normalizing transforms" that would mean the ability to extract fields based on any XSL + XPath that you like?
#11:52:17dbswe have to do this soon, or mleggott will dust off the <marc> patch for custom display configuration :)
#11:53:51dbsmiker_: (actually, in all seriousness, he indicated he would be willing to contribute several thousands of dollars to fund "the right way" of doing that)
#11:56:37jeffmiker_: no metarecord source map that i could find under the standard Bibliographic Record source, but I did find mmrsm under all sources
#11:59:12jeffbut i suspect that starting there will exclude bibs with no metarecord mapping... sound right?
#11:59:19miker_dbs: for "set of normalizing transforms" I post-extraction transforms, like naco-normalize, un-double-space, to-lower-case, etc ... stored procedures that act on the strings after they're pulled from the marc. right now, the normalizations are all hard-coded in perl inside the ingest app.
#11:59:40miker_jeff: right ... you won't have access to the bibs at all
#11:59:45miker_jeff: lemme look
#12:01:27miker_jeff: because Metarecord has to jump through the source map, try using that instead ... I'm not certain that it will do what you want, though
#12:03:55miker_dbs: the functions in 020.schema.functions.sql are generally what I'm referring to for normalizations. things that could be applied to an extracted field. you'd register those with any entry in config.metabib_field ... naco for search_field=t, normalize_space for display_field=t and facet_field=t (for instance)
#12:06:33miker_I can imagine more as well ... getting to that point, when combined with a PG-version-specific varient (xml stuff) of the (commented out) functions at the top of 002.functions.config.sql would allow us to move ingest entirely inside the db
#12:09:24jeffmiker_: yeah, i don't think that mmr or mmrsm will get me what i need, so without any link to the metarecord from Bibliographic Record, i'll just pass this up to someone with db access. thanks!
#12:16:27miker_jeff: well, my uncertainty stems from the code that attempts to build complex multijoin queries, but if it ends up doing what (in this situation) I would expect, then it will give you what you want. The problem is you've got mixed inner and outer joins in the metarecord case (SELECT blah FROM biblio.record_entry b LEFT JOIN metabib.metarecord_source_map m ON (b.id = m.source) INNER JOIN metabib.metarecord mr ON (mr.id = m.metarecord) WHERE mr
#12:19:42jefflooks like your message truncated at "WHERE mr"
#12:20:11miker_SELECT blah FROM biblio.record_entry b LEFT JOIN metabib.metarecord_source_map m ON (b.id = m.source) INNER JOIN metabib.metarecord mr ON (mr.id = m.metarecord) WHERE mr.id IS NULL;
#12:20:18miker_is what we want, but in other, similarly structured situations we want the inner join to trump the left join, and so ... we need to test) ... (just for a more detailed explanation ;) )
#12:21:21jeffthis whole thing is them further complicated by the fact that i'm really trying to limit it to bibs with attached volumes/copies. :)
#12:21:34miker_heh
#12:21:56jeffi'm going to stop worrying about it (and by that, i mean i'm going to worry about other things) for now
#12:22:07miker_the "do we let you effectively write sql, or make it marginally simple to use" delema strikes again
#12:22:47jeffright. :)
#12:24:40eby_ has joined #OpenILS-Evergreen
#12:43:58dbs looks at MODS stylesheet generating valid UTF8 output in SuperCat, and MODS3 / MODS32 stylesheets generating crap UTF8 output in SuperCat
#12:44:48dbs notes that MODS output does not have an explicit "encoding='UTF8'" attribute in the XML declaration, while the MODS3/MODS32 do (due to corresponding XSL statement)
#12:45:27dbs also notes that xsltproc generates valid UTF8 output for all three, casts a gamy eye at XML::LibXML / XML::LibXSLT
#12:49:21miker_dbs: casting that eye at the perl bindings?
#12:50:14dbsyeah, ish. although right now I'm going to cast it at lunch
#12:50:23miker_good idea
#12:54:11miker_dbs: not that oclc is the be-all/end-all, but http://www.oclc.org/bibformats/en/2xx/245.shtm says 245 can't repeat ... looks like 880 or 242 and 246 should be used instead, but IANAC
#13:07:14dbsmiker_: yeah, but this is common practice in non-English countries
#13:07:43dbsUANAIC (You are not an international cataloguer)
#13:10:01dbs(heh, and for what it's worth, it looks like even MODS3.3 doesn't pull in 880 fields, while it handles duplicate 245s just fine)
#13:13:09dbsbut mapping the second 245 to a 246 31 looks like the right sort of way to handle it, maybe
#13:13:18dbs : blah blah blah
#13:15:31jeffi sometimes wish that reports could be set (per report) to not output anything if zero rows were returned.
#13:15:36jeffthings like cataloging QA reports
#13:15:37miker_dbs: well, right, I'm thinking specifically about usmarc/marc21
#13:17:25jeffwhere it would be nice to have output in a folder, or email about a report be the exception that said "hey, something's wrong!"
#13:17:31jeffand no news would be good news. :)
#13:18:50granthpal has quit IRC
#13:20:57granthpal has joined #openils-evergreen
#13:25:22atheos has joined #OpenILS-Evergreen
#13:26:25eby_ has quit IRC
#13:26:25sarabee has quit IRC
#13:26:25miker_ has quit IRC
#13:27:52granthpal has quit IRC
#13:29:11granthpal has joined #openils-evergreen
#13:48:08eby_ has joined #openils-evergreen
#13:48:08sarabee has joined #openils-evergreen
#13:48:08miker_ has joined #openils-evergreen
#13:48:41miker__ has joined #openils-evergreen
#13:49:34miker_ has quit IRC
#13:49:41miker__ is now known as miker_
#13:51:36sarabee has quit IRC
#13:51:36eby_ has quit IRC
#13:56:55eby_ has joined #OpenILS-Evergreen
#13:58:40sarabee has joined #openils-evergreen
#14:55:56granthpal has quit IRC
#14:58:54lisppaste6 has quit IRC
#15:00:48dbsas an aside, people were pretty surprised that 1.4 is out; I guess it's getting close to time to announce that eh?
#15:03:28granthpal has joined #openils-evergreen
#15:03:52miker_haha
#15:04:16lisppaste6 has joined #openils-evergreen
#15:04:54dbs(although - did we roll a package with my fix for my "can't view report output" bug?)
#15:06:04miker_dbs: didn't see that one, so ... I did not
#15:06:49dbsI thought we might be holding off to see if any other bugs turned up
#15:10:50phasefx finishes scrollback, "Btw, stating the obvious here, I see nothing wrong with the pull and parse MARC in the details page thing.. but it's not good for display in search results
#15:12:23miker_phasefx: agreed. but HoH replacing mvr would be good on both, IMO ... for marc-on-detail-page, I'd rather see client-side xml chopping happen, but that may not be practical
#15:13:40phasefxHoH would be good for switching bib formats under the hood too, if folks were so inclined to try that
#15:13:52dbsphasefx: the problem from this side of the fence is using <marc field="245" subfield="a"> rather than <div class="marc 245a"> or something like that - just XHTML friendliness
#15:14:21dbs(IIRC correctly, haven't looked at UPEI's implementation for months)
#15:14:31phasefxhrmm, I thought xhtml would let you use madeup tags, and just rely on CSS to style them
#15:14:44dbshah!
#15:14:54phasefx is obviously isolated in the mozilla world :)
#15:15:00dbsmaybe <upei:marc> :)
#15:15:01phasefxand not really a web developer
#15:17:11eby_ has quit IRC
#15:20:14miker_phasefx: it (xhtml) does, but that's not well supported in browsers
#15:20:34miker_ffx2 can sorta/kinda do it, IE 6/7, not so much
#15:21:30phasefxmm
#15:21:49phasefxthat's a non-verbal noise, not a browser abbreviation
#15:27:08dbs reads the XHTML Modularization spec a bit, doubts that anyone intends to put together an XHTML module to actually conform
#15:28:30miker_I attempted to make marcxml work via pure-css
#15:28:35miker_back in the day
#15:28:46miker_there are just some things I couldn't get working
#15:28:52miker_but it "works"
#15:29:16miker_there are just some box-model hacks that have to be hard-coded, I think
#15:29:27miker_"works" in ffx
#15:29:46jeffwould it be correct to say that the current "shelf browser" sorts only on call number?
#15:37:24miker_jeff: yes, it's a virtual shelf, sorting all call numbers together. tie-breaking is (more or less) location, then order of insertion
#15:37:34miker_location being owning_lib
#15:52:40jeff"more or less"? :)
#15:53:49miker_back to opac, another big advantage to HoH is speed -- mvr is relatively ML expensive, it parses the marc xml and runs a pile of xpath ... in perl
#15:56:09miker_jeff: well ... see: http://svn.open-ils.org/trac/ILS/browser/trunk/Open-ILS/src/perlmods/OpenILS/Application/Storage/Publisher/asset.pm#L201 (and the methods that follow, cn_browse_pageup and cn_browse_target)
#15:56:17jeff$ fortune -s | sed 's/$/ ... in perl/'
#15:56:21jeffThere will be big changes for you but you will be happy. ... in perl
#15:56:26dbsand back to "arbitrary XML elements in XHTML" - http://www.w3.org/TR/xhtml-modularization/conformance.html suggests that there's a a fair bit more to it than just dropping in some new element ;)
#15:56:27jeffSlow day. Practice crawling. ... in perl
#15:59:49miker_dbs: indeed
#16:28:37_dkyle_ has quit IRC
#16:39:57atheoswhere is the public log for this channel?
#16:41:53dbsatheos: http://evergreen-ils.org/irc_logs I think
#16:42:10atheosthank you
#16:42:19dbslinked from http://evergreen-ils.org/irc.php :)
#16:43:59atheosugh, no search feature. hehe
#16:44:21phasefxgoogle may have some of it, the searchbox on the website
#16:44:30atheosah. good call.
#16:44:42phasefxthough I recently changed the url scheme slightly on them
#16:45:18atheosno luck. I was having a discussion in here about a sip based captive portal. anyone remember giving me a recommendation?
#16:45:30dbsatheos: I could grep my personal logs
#16:45:46phasefxbluesocket is the only one I know that works with EG
#16:45:56phasefxheard good things and bad things about it
#16:46:04atheosthat's the one. I know it was blue something
#16:46:29atheosour board wants a captive portal that authenticates against evergreen
#16:47:01atheosand they really want something we don't have to pay for :P
#16:47:57phasefxthere may be something open source out there that intercepts http traffic, but I doubt it would support SIP
#16:48:25atheoswe have a captive portal that will authenticate against Radius
#16:48:44atheosand I can dream up a way to integrate that into Evergreen, but I don't have time for it
#16:55:47jeffheh
#16:56:03jeffreading scroll, i was just thinking "i wonder how hard it would be to do a sip backend to radius"
#16:57:27atheosopenradius will read from postgresql, and allow you to define the table names and structure.
#16:57:41atheosschema for evergreen would have to be altered a bit though
#16:58:57miker_atheos: CREATE VIEW :)
#16:59:05miker_and with that, it's dinner time
#16:59:40atheosthat just went right over my head.
#16:59:48atheosbut, that's alright
#17:00:10jeffatheos: you can create a view in postgresql that would essentially create a virtual table that uses the schema that freeradius expects.
#17:00:21jeffatheos: in theory, i think there might be some potential pitfalls
#17:00:23dbscould be tricky :)
#17:00:24atheos taking notes
#17:00:53jeffatheos: this interests me. it might be cleaner than trying to do radius -> sip -> evergreen
#17:01:31atheosok, well I may not be qualified to do so, but I think my board is telling me to figure this out.
#17:01:54atheoswhich means I can dedicate 8 hours a day to do so
#17:02:37jeffand creating a postgresql view would probably be easier than creating a freeradius module
#17:02:56atheos would rather work on the evergreen/asterisk stuff, but what can you do
#17:03:25jeff"the evergreen/asterisk stuff"?
#17:03:32atheosphone system
#17:03:52atheosjust some work we committed to
#17:04:05jeffyeah, it's making phone calls here right now to notify some patrons of holds:
#17:04:08jeff26 files moved, 104 files remain.
#17:04:11jeffavgtime = 56.4 - expect complete at Mon Feb 2 18:41:25 2009
#17:04:41jeffGRPL adapted some stuff from the voip-info wiki and we took what they had and tweaked it a bit for here.
#17:04:51jeffhere == TADL, part of Michigan Evergreen
#17:05:09jeffi'm working on overdue notifications via phone this week
#17:05:20atheosah, didn't know someone was already that deep into it
#17:05:34jeff(also something GRPL is doing, but we're tweaking it a bit due to how we get the data from the consortium servers)
#17:05:45jeffGRPL uses a Digium card, we're making calls via SIP.
#17:07:02atheosI gotta run Jeff, I'll pick your brain tomorrow on freeradius
#17:07:06jeffsomeone here suggested what we have now go into /contrib/, and i keep forgetting to ask the guys at GRPL for their thoughts on that
#17:07:12jeffatheos: adios!
#17:07:18atheosI'd like to see what you have
#17:07:22atheosif you've published it
#17:07:28jeffnot yet, but check with me tomorrow
#17:07:34atheosok, thanks. adios
#17:07:55jamesrf has joined #openils-evergreen
#17:12:53rsinger has quit IRC
#17:19:42Branflakes has joined #openils-evergreen
#17:50:38jamesrf slaps Branflakes around a bit with patsy-the-scribe
#17:50:45jamesrfer
#17:53:49dbssorry about your college, BC folks :(
#17:55:18jamesrfdbs: what college?
#17:56:10dbsjamesrf: the one that had planned to migrate this summer
#18:00:45jeffdavisdbs: we've still got a college on our schedule AFAIK
#18:01:00jamesrfeh, yeah, we lost one but we still have another
#18:01:15jamesrfstill i'm glad anything that slows the tide of migrations
#18:01:24jamesrfi mean for workload
#18:01:25dbsjamesrf: ah, okay, the person who came to my talk was the one that you lost I guess :)
#18:02:15jamesrfdbs: i'm sure they'll all be clamouring after you guys go live
#18:03:15dbswell, once we go live and finish off whatever bits we can. our contributions have been a lot more limited than we had hoped, ergo slower pace of some development.
#18:03:19dbsbad conifer!
#18:03:47dbsmust head back home now or risk marital discord. sitka++
#18:03:53dbs has quit IRC
#18:15:03Branflakes has left #openils-evergreen
#18:26:09Branflakes has joined #openils-evergreen
#18:26:37BranflakesHi, all.
#18:33:42Branflakes has left #openils-evergreen
#19:13:13djfiander has joined #OpenILS-Evergreen
#19:41:23eby_ has joined #OpenILS-Evergreen
#19:44:52miker_@later tell dbs dbs++
#19:44:52pinesolmiker_: The operation succeeded.
#20:05:11djfiander@karma
#20:05:11pinesoldjfiander: Highest karma: "dbs" (88), "miker_" (53), and "berick" (49). Lowest karma: "my_laptop" (-11), "miker" (-4), and "rhel" (-3). You (djfiander) are ranked 5 out of 276.
#20:05:28djfianderyour_laptop++
#20:06:02djfiander@karma bradl
#20:06:02pinesoldjfiander: Karma for "bradl" has been increased 1 time and decreased 0 times for a total karma of 1.
#20:06:29djfianderso, the pantheon of Evergreen is dbs, miker_, berick, ???, me
#20:11:49kgs_ has joined #openils-evergreen
#20:29:59jamesrf has quit IRC
#20:56:04kgs has quit IRC
#21:00:46dbs has joined #openils-evergreen
#21:01:00miker_dbs: nice catch on my retrieve typo
#21:01:10miker_that would'a been a pain
#21:01:23dbsheh, every once in a while my typo-fetish pays off
#21:07:01djfianderfuck me
#21:07:10dbs takes a pass
#21:07:26djfianderNASIG published a "guide" for marc holdings data: http://www.nasig.org/publications_serialsholdings.cfm
#21:08:11djfiander"Example A (the explanation of the 3rd 853 field" is wrong http://www.nasig.org/publications_serialsholdings_examples.cfm
#21:08:23djfianderI
#21:08:24djfianderam
#21:08:26djfianderscrewed
#21:11:29dbsNASIG doesn't understand serials, they're just a group of people interested in serials
#21:12:03djfianderI think the problem is that they understand serials, just not arithmetic
#21:13:46berickdbs: if it's any use, i wrapped up my slides and files at acq.open-ils.org/~erickson/berick_ola.zip
#21:14:59dbsberick: cool - safe ride home?
#21:15:31dbsbtw, got a trunk opensrf / trunk evergreen server rolling again; I feel back in the swing of things :)
#21:15:48berickdbs: yeah, ride back was just fine
#21:16:19dbsgood, good. added radiolab to my podcast feed, looking forward to listening some time
#21:17:48berickah, cool. they have some filler episodes that are just so-so. the hour-long ones are usually really good, though
#22:02:37kgs_radiolab is great
#22:03:50kgs_ appreciates berick's suggestion
#22:05:00djfianderif you like science podcasts, Quirks and Quarks, by CBC is also good: http://www.cbc.ca/quirks/
#22:06:26berick will have to check that out
#22:11:46djfiander has left #OpenILS-Evergreen
#22:24:07jeff frowns and tries to make sense of some data
#22:24:20jeff(seems to be a major part of my day)
#22:41:40kgs_ has quit IRC
#23:02:21jamesrf has joined #openils-evergreen
#23:40:28jamesrf has quit IRC
< Sunday, February 1st, 2009Raw Log FileTuesday, February 3rd, 2009 >