Our guests
Johannes Bernhardt, Digital Manager, Baden State Museum Karlsruhe (Germany)
Dr. Johannes C. Bernhardt is Digital Manager at the Baden State Museum and previously led the Creative Collections project, which is dedicated to the participatory development of digital concepts and the integration of the museum in the culture of digitality. Read more about his last project here.
Elena Villaespesa, Co-Investigator, The Museums + AI Network, Assistant-Professor Pratt Institute (United States)
Elena Villaespesa works as an Assistant Professor at the School of Information, Pratt Institute. Her research and teaching areas of interest include digital strategy, data analysis and visualization, and user experience evaluation applied to the cultural and museum sector. She has previously worked as a Digital Analyst at the Metropolitan Museum of Art and Tate. Read more about the ‘Museums+AI’ network here.
Read more: https://wearemuseums.com/reframe-collecting
Uh welcome uh and hello to our first session of uh reframing collections uh which is a dialogue series uh in cooperation between uh the Icom com call um uh the commit for collecting and the we are museums Network and uh it’s a new initiative uh and it’s our third session
Today we already had a session on quering the collections on diversity as a focus for collecting and the whole program is dedicated uh to discussing collecting practices finding new ways for interaction to engage um visitors in the collections and it’s going to be a monthly series uh um with with um I
Think up to six or seven um talks and uh today it’s it’s my pleasure to host the third session on digital collecting um and uh this is a very interesting topic IC for me especially I’ll tell you when I introduce myself but with the digital also with Corona and all the digital
Transformation um uh digital tools and possibilities uh come more and more into the focus of museums and we’re going to explore today the potential of digital tools and approaches for more participatory approaches and participation in in general and also uh uh see how we can use it to uh to engage
People um with our collections more and we’re going to look at digital tools and Ai and participation uh and all um in relation to to collections um I am your host I’m Christiana Lindner um and um I’m um I’m working at the Bon State Museum in
Germany uh uh in a digital project that uh I I’m leading the project Museum x. that’s dedicated to create a digital Museum platform uh to um uh to Foster dialogue and rapid response settings and also rethink the museum experience in the digital world a little bit uh and that’s why I’m specifically interested
In this topic of of digital tools and participation and how we can make the museums more approachable with with it um I have invited two fantastic guests today uh to discuss with us uh uh Johannes Berard uh who is the digital manager at the B State Museum and also
My dear colleague uh um and and Elena Villa espesa who is an assistant professor at the pre Institute in U New York and is co-investigator of the museum and AI Network um and I’m going to uh introduce them longer before their talks um and uh we’re going to spend the
Next one and a half hours uh discussing um but first I think um before we start it would be nice nice to have a quick introduction round for everybody to meet and and then uh Elena and Johannes will present um and talk to each other because uh this is our format that we
Would like to have an intense discussion between our two guests and then we’re going to open the discussion for the whole um uh uh for for all of us and have like a campfire discussion and in the end um I’ll I’ll wrap it up um and
At uh 2:30 I hope we’re going going to U have had a very intered interesting and intense uh discussions about digital collections um I’m I’m I have to say that we’re going to record the session I hope this is okay with everybody um the session will also be released uh uh
After so we can all rewatch it um and um and reminisce also uh but um so let’s start I would say um with a a quick round of introduction and uh since uh I think we’re joining from all over the world it would be cool to say uh who who
You are uh where are you joining us from maybe what the time is so we can know who is uh in the middle of the night and needs uh coffee or something and also what your connection or interesting uh interest um or link to our top today’s
Topic uh is that would be really cool maybe just a quick round to so we all know who we dealing with to quotee The Godfather or something so uh maybe I’ll start so I’m Christiana um I’m joining you from uh Germany uh where it is 1 pm
And uh it it did Snow a little but now it’s training but and my my connection to the topic I already said so um I hand over to maybe Danel hi my name is Daniel I’m the chair of kol um uh but also acting director of
Imag IC I’m based in Amsterdam uh it’s 8 over 1 p.m. at the moment uh it’s raining here already the whole day um so it’s it’s a very gloomy day um so I’m happy to be with you and to see some happy faces to bring some
Sunshine into the day um and for me collecting well as chair of of of comco of course all the different aspects of collecting are interesting uh for me but especially the whole development in the digital um sphere is is of course top of Mind especially after last year but it’s
Of course already a development that is going on for so many years uh and my specific interest is also in the ethics of digital collecting thank you danela do you want to uh nominate somebody else should we do it like that or yeah sure sand thank you Danielle hi everybody I’m
Sandro debono I’m an academic at the University of Malta so is the Mediterranean here it’s uh 10 minutes past one and it’s it’s it’s good weather as expected from the from the Mediterranean although don’t take it for granted um I’m an ex Museum director so um Collections and collections
Development has been a topic which I’ve dealt with not just as re as far as research is concerned but also from a practical point of view and I’m here to discover new things to learn and to exchange ideas with with the rest of the of the
B uh yes I have to pass the word sorry to PARTA I think you need to um uh start your yeah you’re muted yeah hello my name is Martha I live in Barcelona uh the weather today is really sunny it’s also one o’clock here 1 pm and uh I’m
Working as a volunteer for two organizations one is REM is the Spanish network of museums and digital strateg so uh Elena is one of the speakers in of in our conference next march uh and also p is involved in the event as in the scientific committee so
It’s it’s good to see some familiar and Spanish faes too and um I working also for another organization is the Aon icon is the committee for Audi visuals new technologies and museums and and social media Museum sorry so I’m working as a volunteer but looking for a
Job and you know uh I would like to be involved in digital projects in museum so uh learn about Digital Collection is is interesting for me so that’s why I’m here so uh Floria hi everybody happy to be here with all of you um my am fencia from
Argentina VES it’s raining here too um I work I am currently working at the national EV Museum in the department collection and I am a young board member of comco so that is why I’m happy to be here and maybe Elena next one so introduce myself then yeah maybe
So uh my name is Elena I’m from Spain um I live currently in New York um as you can see my windows uh it was pretty dark like um a few minutes ago it’s 7 12 a.m. here um um so yeah I will be speaking today I
Am assistant professor at prad and um I’ve been working in museums for the past like 12 13 years in digital projects uh currently focusing on AI so I say next one H Chelsea hi everyone I’m from Taiwan and uh we are the colleague of comco with
Young member and I’m very happy to meet you guys and we are uh looking for some Concepts about collections to collectings so the concept of transformation uh to museum is more like active so it’s what I’m interesting in the digital and some thing like collection to transformation uh this is what I
Interesting so very happy to meet you thank you and the next one may be Alice hi everyone um this is my first one here um from Swansea Wales it’s very rainy here um I’m a master student but I’m also a volunteer transcriber with the welcome Institute um into an online collection
So I discovered this through Linkedin and was very fascinated by the topics I’m here primarily to just observe and see how I can utilize the online collections into my volunteering oh sorry um volunteer um silk Silka s you’re muted um I think now she’s Frozen yes yes
Maybe we quickly move on yes maybe uh Aida is next okay hello my name is zaa and I from Lithuania and I live in Tel and work in this Regional Museum called s magician Museum ala we have more than 75,000 exhibits there and um I’m interested also in digital projects and
How to to um give more attention to exhibits that we have in our Museum and it’s we have um much snow in Lithuania right now it’s but it’s not cold so I give word to rasa from Lithuania hello everyone uh yes I’m Lithuanian but I’m actually not based in
Lithuania I work in the Netherlands at the Netherlands Institute for sound and vision uh as was already mentioned Netherlands is having a very gray rainy day so I’m always happy to be here as well zoom zoom is much more cozy at the moment um and uh I primarily work on European
Innovation and research projects and I’m particularly interested in making Museum collections more diverse and polycal uh and more engaging to to various communities that we perhaps have not been engaging with using digital and analog methods and I don’t remember who else hasn’t introduced themselves maybe Mara and Johannes are
Left oh and Marina TR sorry yes okay so I go forward my name is Marina I’m from ol Germany I’m working for the museum ol and the project manager for NEX museum. which is a digital platform for co-curation and co-creation we are um part of the fund digital of the German
Federal cultural Foundation um and it’s a joint project with the NV Forum dorf and I’m very interested in um yeah rethinking uh the collection um we are working with new art and Next Step will be um to focus on our collection in the museum so I’m pretty
Curious and it’s nice to meet you all it’s my first time here it’s actually my first day um in this community I’m sorry but I think it’s only Yan’s left right and Alina and uh konga and Sil when she’s back yeah and who shall I yes okay my
Name is Johannes uh um I’m working with Christiana at the B State Museum and I’m the as as Christiana said digital manager um weather is catastrophic uh in K at the moment gloomy as hell um so it’s good to be here and I’m looking forward to the
Discussion hi I’m Alina uh I’m in Berlin and we have in this moment a bit of Sun so you can see it on my half of my face uh it’s it’s not raining but it has been very gray the last days and uh I’m very
Happy to see you here I’m based in the Jewish Museum Berlin and part of the com call family so I’m also interested in every aspect of Collections and especially right now with uh us not moving be able moving around this aspect of digitalization of collections is becoming increasingly important I think
For not only for people who work with this professionally but for everyone in the museum so happy to see you okay then maybe Sila you try again yeah thanks I why I think it’s working now yes okay because I was Frozen um so my name is ZKA I’m working
In this joint project which is called Museum 4.0 with Christiana and um in our project there are seven museums working on digital tools and I’m in the organization team and my role is the communication and education and so I’m interesting in everything about we will talkk you thanks then Elizabeth has
Joined us uh would you like to quickly introduce yourself um good afternoon everybody I’m based in Berlin I’m Elizabeth marovich and I founded a few years ago uh still the only platform on every screen smart TVs and know showing basically your collections um as uh we’ve been focusing
Only on the collection because we were seeing that all the museums were focusing on their temporary exhibitions and uh we were interested in their collection and we’ve been doing very unique films on each work and today we are uh opening the the platform to every
Mum to become a hub where a viewer can have a true art experience whatever it is with art video artist films interviews and even entertainment films all around art thank you welcome um we still have uh cona uh left if you if you want to to uh
Switch on your video and say say hi um if not it’s fine also okay then maybe um uh we we start uh our discussion section it’s really great to have such a diverse group from all over the world uh here and I’m I’m already very curious what we’re going to hear
And discuss in the next hour um well as I have said uh um the digital collect uh digital um tools and digital approaches to collecting are becoming more uh important day day uh daily I think also as Alina put it uh and it’s it’s very interesting uh that like the digital
Sphere in general has a great great um um potential for participatory approaches because in like in the digital it’s it’s it’s inscribed like uh people produce their own content on the internet uh they they uh meet and share things they co-create uh things and so in the digital sphere a lot of things
That we discuss in the museum world for being more inclusive diverse and participatory it’s already happening so it’s really interesting to look uh what we can learn and use uh in the from the digital um to um to enhance or create more um inclusive Collections and uh
Create new ways into the collections and uh I’m very happy to to announce our two guests now again I think three times for introduction is also a good good count so um um johannas and Elena are both going to uh do a short presentation now
For 10 minutes each um and um and then discuss with each other but we decided um before that um if you had any uh questions after each talks to clarify or anything so there will be time to ask these questions but for for the discussion uh I would ask you to wait
For our campfire sessions um if that’s okay so Johannes is going to start and um um it’s as he already said he’s my dear colleague at the Barton State Museum but before he went into the museum world he spent 10 years uh as a researcher and teacher at uh different
Universities in Germany and uh at a bon State Museum he was responsible for for the creative collections project where he was in charge of uh creating uh uh digital tools um in participatory methods together with our citizens and he’s an expert in uh product design methods like design thinking and participatory methods and
Um he’s now the digital manager and responsible for um the whole digital Enterprises Endeavors of the museum and uh the digital strategy and um uh um well I I know a little what he’s going to talk about but I think I’ll leave it to him now to to introduce his topic
Johanes the floor is yours so uh thanks for the introduction and um thanks for the invitation I’m thrilled to be here um and the task for my talk and it will be around 10 minutes uh was to say something about the connections of participation digital collections uh what is going on in our
Museum and how all this leads to um artificial intelligence because that will be the kickoff for Elena’s uh top so uh you have a uh a first impression what is going on I share my screen um and don’t be irritated I’m uh in this call two times because I’m using several
Computers uh um uh it’s always me can everybody see it does it work yes it works yes okay okay so I start right away um uh here you can see uh C Palace um in southern Germany um the um home of our Museum it’s a cultural Historical Museum with a large
Collection 500,000 objects um and uh everything I will say um in the following is um uh um conditioned or or has as a starting point that the museum um will be uh renovated and remade um in the next five years around five years uh because um the museum was completely destroyed
In the second world war and rebuilt and since then the infrastructure has been more or less the same so it’s pretty dated and this situation that we um are looking for um or we are going into a renovation process gave the unique possibility to rethink the Museum from
The ground up uh so that’s the starting point um so uh we developed in in a long process uh new Museum concept and the the core idea of this um Museum concept uh was and is that the visitors um should become active users of the museum
Um so as you will see at the end of my talk we are starting to go uh one further step but this is the core idea and um to do this um we developed A New Concept to present and to um uh to access our collections um so we
Opened in 2019 our first prototype exhibition um about the archaeology in Bon objects that were found uh in the Bon region um uh mainly um from prehistoric times um and this prototype has in fact three parts uh the first part uh of this exhibition you can see
Here is uh showing some high light objects um very very uh uh yeah stenograph um I’d say and then you have this part here we call it the Expo um uh where you can um explore and research the objects and then there’s a third part you can see
Here um it’s called the Expo um lab where you can um experience objects via a virtual reality in their original context and experience small stories and um yeah experience the material directly but the important part for um uh the access to the collections is the xque
The image you can see here um and uh this is designed like like a laboratory where you have very very many objects they are on a small place um maybe 1,500 objects um you can see here can you see my my mouse uh cursor here does it work
Yes yeah you can see here draws um and all this is filled up with objects there are no um object signs or or anything uh if you enter this exhibition you get a smartphone we call it the Expo phone um and Via augmented reality um you can uh identify objects directly with the
Smartphone and then put your um smart platform on one of these media tables and the collection uh of your objects will be um uploaded to these tables and there you can do a lot of things play games access deeper um um deeper knowledge about these objects and in the
Background is a digital catalog um and uh a new thing really interesting opens up a lot of possibilities we have now um um a user ID uh and replace the classical entrance entrance ticket to the museum by ID um uh user IDs uh that are working for one year and are working
Like a library card or Library ID and um if you use the dtic catalog and your personal user ID you can um search objects that are interesting that are in this collection and you can order them like in a reading room and then they will be taken out of the uh of the
Showcases and present it to you directly and if it is possible maybe a Flintstone or something that isn’t uh um too fragile you can take it into hand even so what we are trying to do in this exhibition and with the access to the to the collections is uh to use analog and
Dialog um uh connections to to to to access the objects of our collections directly so um chisan was uh talking about this creative collections project um that is kicking off um uh at this point and there we were um uh trying to think digitality uh and the digital um
Together with participation and as Christiana said one of the keystones of this project is that um the digital is already participatory uh if you want it or not it is uh so what we did in this project was was um to uh establish a citizen Council of the Museum of around
50 uh people as diverse as possible and we invited them to um develop together with us and starting from the infrastructure we already have um New Concepts new digital Concepts that would be interesting and I show you some random images um of these workshops we we um organized in our Museum there were
A lot of them and um we were using design thinking methods so the citizens could de develop their own projects and their own Concepts and uh there came a lot of prototyping material up in this process and we combined this um uh this approach with um the format of Museum
Camps it was somewhat inspired by what was what Nina Simon is doing in in was doing in Santa Cruz and colleagues in Amsterdam um with this form but we um gave it our own spin uh and organized it as an open bar camp where everybody was invited to um to participate and to
Discuss topics that are interesting uh um and to develop the Museum of the future so again some random pictures uh um up on the left side you see one of the plary sessions then um on the right side the session board where um all the topics and themes were collected
And uh all these topics and themes were proposed by the citizens themselves so we um only um um gave the framework uh for this bar camp and open conference format and um the citizens could decide by themselves what they wanted to discuss and it was pretty mind-blowing
What came up in this process so um next we went uh some step further and established um in uh um the museum at the marketplace in the City Center it’s part of our Museum infrastructure an open space um uh to to make it easier to organize participatory
Events and uh we called that one the museum X um and uh we called it Museum x uh with this variable the X uh because we didn’t want to give uh ansers but to discuss with everybody who’s interested um what the Museum of the future should
Look like and um since we analyzed um the whole process um of our participatory events and workshops and camps and so on and had a pretty clear pictures about the needs and interests of our citizens uh we organized in the museum x hackathon uh with 50 participants um that were developing
First uh prototypes functional prototypes of uh what we were discussing uh with our citizens and some of the projects are still ongoing Christiana and I um are working on this uh on this material a lot um and I show you maybe uh two complimentary tools um that grew
Out of these processes one is um um the the web app um uh your gift D geng um which offers a framework a really open framework um where citizens visitors users of the museums can collect objects um in the museum um and frame them as a
Gift for someone special um and put in audio files and audio messages and personalize it it’s a very intimate tool um we developed in cooperation with a British uh um um artist Collective it’s called Blast Theory and we we um uh translated it and developed it a little
Bit further because it fit perfectly in what was going on in our citizen processes and another project um which combines digital and um um participatory things is the is the app my object um that uh we implemented in cooperation with the home World forum and the museum
For 4.0 um um and network uh that was mentioned earlier a really interesting tool that gives another access to our collections because that is what you can see on the left side you can swipe through our collections like in Tinder uh and then if you have found an object
That fits your your interests um you you have a match and then starts a dialogue like in WhatsApp or telegram or or whatever messenger you use um and the object starts to talk to you and uh invites you to visit it in the museum and uh actually these dialogues in the
Background are scripted and um um we organized Christiana was heavily involved in this um we organized a participatory process within our Museum so a lot of colleagues of all departments were involved in writing these um dialogues and um a lot of uh citizens too were writing dialogues for for um
Uh this uh this app so we have a wide spectrum of different viewpoints opinions approaches it’s really interesting I think um a really yeah polyphonic uh Endeavor um and so I I’m I’m I’m reaching I’m I finish in a second um um so we were
Reaching uh the AI um topic um in this process because it was pretty obvious that many of the needs and interests of our citizens could be addressed perfectly um if you approach the um artificial intelligence um field uh to personalize um um the materials we offer to our
Users um and we have now a project it’s it’s starting these days uh yesterday was one of the kickoffs to do it and to um um develop a tour we call at the moment it’s working title I curator which helps our visitors and users um to
Curate their own content and put it into our system uh and so this is what I meant in the beginning we are trying now to develop our Museum concept from visitors to users one step further to active co-creators and uh there comes this other project um inter viiew that
Christiana is uh leading at the moment the museum x0 where we are trying to develop a new digital platform uh that um offers new possibilities for rapid response settings um and dialogue between the museum and the users and the co-creative development and of new experiences it follows more or less the
Same approach but I think I’m a little bit out of time so um I think I stop at that point and uh maybe there are questions because it’s a lot of material thanks for your attention thank you yanas for the presentation um are there any uh any
Questions um uh um um for the presentation um I I already saw one in the chat that I’m I’m going to read uh are are the apps in German language or in available in English too um a question from Ida uh at the moment they are only in in
German um but there are plans and um I think we will go on in this direction uh to have at least an English version um but uh for example the the app I showed um uh your gift um where you can uh collect objects and uh frame them a
Frame reframe them as a gift was originally um developed in English and there for us mainly was the problem uh to get a German version together um but there is already an English version that can be used and in fact it’s an open source project um you can use it right
Away if you’re interested uh so um that’s that’s a thing we are trying to do a lot with all the things I’ve I’ve shown you um to do is to do it open source to have proper documentations um to put the stuff on um uh GitHub and to
Have an own GitHub account for the museum so others can use this material too especially for example smaller museums that don’t have the infrastructure to uh to develop large app projects or digital platforms and so on um Sandra had a question uh uh in in what ways were curators and conservators
Roped in and what’s your experience with that um maybe if if this is to answer but maybe quick quick answer yeah yeah that this question implies a lot I think uh a lot of experiences um I think um but uh that’s mixed B so we have curators that are heavily involved
Really interested and and some are of course not uh so and I think that’s that’s just uh uh the the the the usual situation you you envisage in the museum uh but we have have fortunately a lot of curators that are really interested that are trying and starting to think about
Their exhibition projects by including the app projects for example B app my object where you can have a chat with an object um is something that is also interesting for our um curators um so they are they are thinking a lot about it how can I include it in a in in an
Innovative way in my exhibition project that’s already happening and we were trying with this design thinking approach uh for example when we involveed the citizens in the concept development we tried right away um to um uh to have um members of all departments in our team to um moderate these
Workshops with the citizens so uh there were curators Administration people restoration people from every department and it was intentionally done so so that this would be would have an effect effect on the on the whole structure of the museum and um uh at the moment with the curation I curator project where we
Are opening this uh this um artificial intelligence box we are we are involving all our curators because it’s um their domain we are entering uh with this tool um maybe a quick uh uh answer also uh is there a chance to get more about the I to know more about the I curator
Project asks yeah it will it will um in fact um there will be in some Weeks end of March I can send a link later or Christian can maybe do it there will be a network meeting uh in in our Museum a digital one everybody can participate um Elena
Will be participating too I’m really happy about that because we are trying to um there there’s already this network in the english- speaking word Elena will talk about it in a minute I think and we were trying to bring together um in the German speaking World um museums and cultural institutions
That are trying to do something with um artificial intelligence because it’s a lot going on and it’s hardly visible yet um and so we thought it’s a nice idea to have a network uh and to make the stuff visible so um there we will discuss also
The aurator thing uh but the project is just starting we are organizing with our um um Partners in Amsterdam at the moment uh but we will publish material soon I think uh so you can have a more insight in what is going on and what we
Are doing and there will be probably a lot of events too okay thank you very much um I would say uh if there’s any more questions maybe to save them for for the campfire discussion later um thank you Johannes for introducing um like the uh a range
Of digital tools and and our project um and uh now it’s it’s my uh great pleasure to introduce uh Elena our second guest again um as she already said uh she’s an assistant professor at Pratt Institute and her research uh focus is on digital strategy digital user
Experience uh and uh she did her PhD uh in in in ler University in the UK and she also worked as a data analyst for two big players in the museum world the Tate and the Metropolitan Museum and she’s uh the co-investigator of the museum and the AI project um uh in the
UK and the US and uh she’s going to uh tell us now uh uh uh um a a lot I think about uh museums and Ai and her work and research in this area Elena the floor is yours thank you chrisan let me share the screen
Oops all right so you are looking at my screen right now right yes okay perfect so yeah thank you for the invitation as well to participate here in in this discussion um I’m going to go briefly over the project this museums and AI Network project that we did in 2019
Uh myself and also Dr un Murphy from um goldsmiths university of London um so here you have uh it was funded by the Arts and Humanities research Council in the UK and we have um some museums who partner from the beginning and also for the funding application and then
Others join during the project so we had um the med the National Gallery the American Museum of materal History and then of course the partnership between our universities so just to give you a bit of context this is one thing that we did um at the beginning of the project was
To map like okay how many projects what type of projects Mims are doing in the area of um AI artificial intelligence is a pretty big concept and it’s very hard like to get like a good classification of topics but um we did our best here in
The topics that we see we saw museums using the most um so here there are about I think it was like 70 projects that we found in the museum sector of museums using Ai and as you can see in this timeline the the growth starts really
Like the last three years and I have continued and this um list of projects and now I have over a 100 um and of course in the past like year there has been a a lot of work on that area too you can also see that there is a huge
Interest in the museum sector on machine learning and specifically uh computer vision which I’m going to focus a lot um on today um the project focus on how to use AI for H Museum data we focus on visitor data and also collection data but for the purpose of today’s talk I’m
Just going to talk about the latest one so um at research questions were like what are the opportunities and challenges to apply AI Technologies to the collection data um how Comm minimize the algorithm biases to interpret their collections um something very important with the lack of diversity in the museum
Field and also in the AI field be reflected in the outcomes of using these Technologies and also you know what are the implications of museums engaging with big companies to uh develop um the different outputs um so yeah as I mentioned um we started small but then uh the project
Grew and involves six universities 50 experts 50 museums and also we did two big events uh which involved uh 200 members of the public uh so here you have some photos of who these people were and our Workshop work so we did one event in London at col Smith and then
Another event at my University at Prat Institute uh the two big events were um at the barban center and the second one at the Cooper okay with I will add later the the link because you can watch um the uh the talk and the discussion uh on
YouTube so um during those events and workshops several case studies were presented um so I’m going to mention um very briefly those projects and then I will uh talk about the conclusions of um the project so for example the welcome collection is using uh machine learning and computer vision to identify you know
Different elements in their artworks and be able to link and find related content in their collection uh the Princeton University Art Museum is using natural language processing uh to basically identify different entities in their text so then again they can create relationships um in in in their materials and Harvard art museums they
Have these um I added the link so you can find more information they have this website um where they provide the data of five different computer vision systems you have here Amazon clarify imaca Google and Microsoft they apply it to their collection and they provide the results on this website so it’s very
Interesting because you can see the results from all these five different computer uh systems and they created uh subject tags but also labels for objects uh the med has been doing a lot of work on um Ai and they have been using as well different computer vision
Systems and they have also taged the collection manually adding subjects to 300,000 objects um and they are not using that data to train uh algorithms they have also done several competitions online involving the data scientists uh to discover and think of new approaches uh to represent the
Collection uh Moma who was also involved in the project presented a case study about how they use a computer vision to analyze all photographs in their archive they were all exhibition photographs so what they did was uh use the algorithms to identify the artworks in those exhibition photographs and be able to
Link that to the online collection so if you navigate navigate now the photographs archive you can click you can see this um um purple like uh Dots here you can click there and then um access the online collection the specific artwor there um another partner in this project
Was the science museum group in London and they again use uh algorithms to create tags they are doing a huge massive like digitization uh project um like digitizing I don’t I can’t remember the number but like hundreds of objects a day and of course those need to be
Catalog and they need a description and everything and that takes time so they’re using um AI Technologies to to create subjects that will allow users to search uh their collection um so some of the conclusions of the project and um what we talk about is that of course AI presents new ways um
To you know showcase The Collection to provide access for users and as you can see here in the quotes from some of the particip ipants they talk about you know how they can create and this additional metadata or new metadata in some cases that um will allow users to discover our
Their objects online and here in this second quote it’s interesting because um they were talking about how um you know in museums we have this um specific terminology and you know our vocabularies um to describe artworks but when using computer vision and other systems we were able to describe
Collections in different ways using terms that were not in basically the museum um um like vocabulary so here’s an example this is from the project I presented from the Harvard Art Museum so you can see for these um still life that the different computer vision systems brought
Different keywords uh so many of them like were were able to to identify that you know it was a watermelon and it was like fruit it was food but then you can start seeing some uh differences and even some interpretation of you know the artwork like um I think was yeah imaga
Was saying it’s sweet H delicious it’s ju juicy and clarify was saying that this was like healthy again juicy and so that was interesting because of course that data was not in the uh Museum collection database and same here uh with the as I mentioned they have taged manually their
Collection um and you can see those Museum tags here in this table for these artworks um and those included you know like the gender of the person represented on the artwork uh whether it was a portrait or a landscape but it was interesting from the computer generated
Tex to see that they were able to identify some um um clothing elements in those artworks you can see here like dress garment or hat that it was not described by um the curatorial or the digital Department um however one thing we learn in the project is that all these
Technologies are still very uh experimental um there is this tension still of you know um data is not 100% accurate um how we’re going to display data that is not correct in our website so there is this tension and this internal debate about you know how to um
Use this data it shows some advantages to discover and provide new uh information to the collection but there’s always there’s always some risk and we know we’ve seen it in the news how algorithms have sometimes have failed really badly um and another H problem um and that comes to resources
Is that there is a lack of AI expertise inous um museums are currently engaging with big tech companies um but there’s only a few cult institutions that actually have a data scientist in in their like staff so uh that brings a lot of challenges and also even when using
These Technologies how you know you run the algorithm and those things are kind of like becoming easier and easier like getting that data set of tax but how what is the process now to clean up the data and um bring it it to um an app or a
Website um however some museums some of the museums presenting their prototype and some examples of how they are using the data so this is Art Explorer um from the med and this is a website where they’re prototyping uh how to provide an interface where users will be able to
Search using different tags as so you can see here oops yeah oh no okay my mouse is going crazy so you can see here the tags um and then also some suggestic tags in the search box and instead of you know doing that typical search uh using the department
Or the medium you can also uh use um the subject topics to do that and here’s the example from the Harvard Art Museum uh they have this website the triple if project Explorer site where again using different ey Technologies and using the data from those five computer vision systems they’re generating different
Like interactives like magic message for example you type you know I love cats and it looks for those um words like I love cuts it looks for for example the word Cuts in their images so it provides you a result with the sentence you wrote but um Illustrated with artworks from The
Collection like cuts from uh the artworks but I mean one of the things we know is that uh there’s a challenging uh process working with a i in the whole like data cycle from um when you gather the data when you train it and also um
With the results and how good or bad those are um one of the things we discussed during the project is that uh in the training process you need a lot of data to train the algorithm um This was um jinny Choy from the med who said like
Okay our collection is huge we have 500,000 objects but we only have 600 cuts and you know it’s not enough to train the algorithm and as you can see here you know it the cuts are very different depending of the art period so it’s very hard for the
Algorithm to learn of the different type of cuts in art history um and then there other challenges like um the computer vision systems were very good at you know identifying colors and uh shapes and forms but is very hard to get contextualized and interpretative data from um the usage of computer revision
Systems um for example in this um sculpture of Cleopatra and computer algorithms were able to see that it was a sculpture that it was um you know it was Art um maybe even like you know um the medium but they were not able to identify that it was Cleopatra and the
Same happened with many history IAL and political um um people in in the Mets collection they were not able to identify um George Washington or others and of course the biggest challenge is like it’s not 100% accurate algorithms are getting better and better I had the data from different years uh
For the meth collection and it’s getting much better but still you know you get like huge errors like I got some examples here here from the science Museum uh this chemically fertilizer it says that is a musical instrument a rattle um like gender challenges and you you know repres and like that’s very
Challenging to display on a museum collection like I’m me identify as a lady and then I like this one that um the algorithm thinks is a a diaper bag um and just very briefly um one topic that um we covered a lot during the project was ethics um because you
Know uh we’re using data we are using external algorithms we it can’t um it could be risky so um we talk a lot about the need of an ethical framework that you know could respond um to to this new emerging technology and and that’s also you know has been discussed um in
Conferences and I have this quote here from uh Elizabeth Mary the director of the American Alliance of museums that says you know that that you know museums have now this role as well not only to use it ethically but also to uh raise those issues to the rest of the public
And teach the public about it um so I I wanted to share a couple of resources uh with you today so what we did at the end we wanted to be a very practical research project with something tangible that museums could use uh so we published um last year at
Toolkit I will add the link in in the chat so it’s a series of worksheets and that museums can use when planning an AI project it talks it makes you think about you know the data workflow um how to manage a stakeholders and also what are the uh capabilities uh that you need
Uh to undertake this type of of projects um so I have the link here I put it not in the chat and also actually yesterday we published um this paper uh in uh the museum management and creatorship journal I have a link uh to access some free copies we also put it
On Twitter I don’t know how many have been downloaded but I will put the link here we only have 50 copies uh available but this paper focuses particularly on the usage of computer vision on um on museums um yeah I’ll leave as well my uh contact details in case you have any
Questions stuff sharing now Thanks Elena very much for this interesting insight into the AI world and network um since we we landed on point at two um I would suggest that um we we go into the campfire discussion right away but uh to keep a little bit
Of our um uh proposed setting I would maybe ask our two guest guests to start with with questions to each other and then please feel free to just jump in if if that’s okay so um Johannes and Elena do you I’m really sorry for the chaos in my video at the
Moment uh I had to run to my uh sleeping room because I have a construction site in my in front of my window and it became unbearably L sorry um okay so maybe maybe I’ll start with a question uh then uh you can uh uh think
A little of your own um because when listening to you and and I’m I’m I I’m hosting this now not not not as a colleague of Johannes but as also a board member of Icom com call and I’m I’m already uh always interested in um how would you say because the both of
You uh talked about uh digital Tools in in rather big structures and bigger museums how would you say also listening to you Elena about the uh you know for AI you need a lot of resources how would you both say U our digital tools uh um available for smaller museums uh what’s
The potential there or is it is it possible uh for smaller museums to implement these things um Elena do you want to start sure uh yeah that’s um that happens with so many things not only with yeah Ai and um I have to say what is interesting about um AI
Technologies and they are becoming easier and easier for people to use even myself with very limited programming language knowledge I’m able to um like use those Technologies and get the labels and the the subject keywords for for artwork using just images so I don’t need to do any like Python Programming
Or anything so I do hope that those Technologies become more accessible for smaller museums to use and I also hope you know that um one of the things we actually discuss in the paper that there is a need for collaboration in the museum sector to come up with um some
Guidelines and collaboration so um you know um museums help each other as we’ve done with um other topics so um I hope when museums use those Technologies they can uh release you know their code and in open sources platforms so smaller museums can use those Technologies yeah may I
Yes I would like to to to to underscore what Elena said I think um what is going on at the moment uh you can see it in in in uh many fields is that uh uh the opening of the museums uh is changing also the uh approach um to to handle um
Digital tools and um uh platforms and so on uh that step by step uh open source culture and the open culture um um becomes more popular and it’s maybe a generational thing that younger U Museum people um are getting in charge there are a lot of Young Museum directors that
Are pushing open source approaches and I think it will become stronger and stronger in the future and I think it’s there’s actually no alternative to it uh because this old approach that every museum is developing its own app and then you have 5,000 of them and uh not
Even one is working properly that’s that’s absurd uh so and I think um in uh with a view to AI this is maybe one of the most interesting implicit aspects of this field that that no Museum will be able to do it alone um so we we are
Reaching their point in the digital sector where museums have to cooperate if they want to or not or otherwise uh AI approaches will come in from from other sectors um uh and uh be be applied to museums from the outside um there’s now a forc to do it um and uh really a
Need to to cooperate um and I think this really interesting so uh I think it was a great idea in in the English speaking world to um uh to establish an AI Network for museums and we had a similar idea for for uh the German speaking
World to do it um and from right from the start to propose what could be ways to cooperate on chatbots on uh um a computer vision on language processing uh to to bind together different approach Roes and initiatives um so so you can have synergies right away
Um and I think it will happen uh it has to it’s no longer an option it’s it’s it’s it’s necessary are there any ah Alina yes um yes thank you so much for uh for this presentations and which are very inspiring uh I’m I’m very interested in the question of categorization and so
The question which I have in this connection is to both of you how uh the digital devices or the opportunities which is brings with them allows us to work on categorization um because I mean we can we can we have our categories in our cataloges so when we are looking for
Objects we we go to one or another but we realized today that this categories actually needed to be redone and reth thought and um so where are we standing or where are you standing do do you go back to and and think about recategorization or do you work with
What is there now so which opportunities do and challenges we have concerning categorization of objects Ela do you want to start or shall I go ahead go ahead so I I think there actually two ways uh I I think the starting point is uh um really just to state that the
Categories of a museum are usually not the categories of the visitors so or very often uh they don’t fit uh so because in a scientific perspective things are important that are utterly irre Irrelevant for for visitors so um so you have to State this and then I
Think you have to two uh um directions or approaches um the first one is um if it uh concerns the digital you have a database or or something where where categories are are working already and you open that thing up uh and and see what users are doing with it
If you are able to involve them in working with the material in in in any way for example you invite wikim medians to to take uh some thousand object data sets and to work with them or to take a a part collection or several hundred objects and do it with this wikipedian
Crowdsourcing approach for example it’s it’s it’s One Direction the other uh Direction and approach in my view is to use these tools Elena was presenting um to recategorize in the automatic automatic way so that’s maybe one of the most interesting aspects of using artificial intelligence approaches that you can really quickly
Relabel and uh recategorize uh collections uh in different with with a different perspective um so there there are already a lot of interesting um art projects going on for example where you can some you can use right away on the Google Arts um experiments platform where you can choose for example from
The arts and culture uh um a database two objects yeah random for example a Greek waste or um some furniture from from the barck or Renaissance period and the AI is creating a line of objects that leads from one to the other yeah so
You have um a set of uh objects maybe 10 15 20 50 you choose um and the thing is they are categorized and organized in a way that has nothing to do with arttory there are criteria working in the background that are completely different yeah and it may be interesting for
Example for for visitors to do something like that I’ve I’ve seen this in in an exhibition a while ago where users were happy to experiment with this kind of uh uh organization and that’s that’s only a simple example um so you can do much more complex things um and uh use for
Example an image database to uh re-edit uh all the keywords and the um labels of objects yeah I would say that’s a very like like challenging process uh to have that um like yeah categorization and recategorization of and clean up of the data just like those open sources like
Databases that are available you can see museums um even just like the manual input is not like 100% like correct or clean uh to do U data analysis um but I wanted to mention regards to computer vision that um I did um a study comparing at the med the museum taxs the
The ones developed by a digital Department in collaboration uh with the cators um I compared those with the computer vision generated tax and trying to see the overlap you know like how different or similar those are and they are so different it was um I think the museum has about a thousand keywords
Defined um for subjects and computer vision returned about 2,500 so the volume was higher in uh computer vision for um a similar like data base um of objects and there were less than 200 in common and so that shows that it brings a completely different vocabulary to to
The table and museums have to to make a decision of okay what are the what are the keywords that um we are interested to get from uh computer vision systems and how that impacts our cataloging and categorization uh process um because some of them could be very useful for
Missions with no data at all like providing access you know by colors computer vision are very good at you know identifying and like yeah what I said like shapes or objects in in in the artwork but they are very that contextualizing the data um so yeah how
Mums have to make that decision when youer computer vision and for the med some of the keywords were irrelevant like museums computer vision will tag with um for example art or um like things that are already in the system even the medium or um the uh type
Of artwork so yeah that’s um a challenging uh process and and also when computer vision algorithms get better or they change how to uh again modify your collection database like Google removed uh gender related tags I don’t remember it was like two years ago so if you had
Already your collection tag with uh Google computer vision systems how do you change all your you know all your data Danel thank you so much for your great and inspirational presentations I was just thinking about the the theme of this series which is reframing collecting um and of course in first
Instance you would immediately think about a lot of the the things that we discuss now is reframing collections but maybe not the collecting process and I’m just curious to hear from both of you if working in this way will also um change our way how Museum will collect in the
Future um absolutely um um maybe I can pick up one point of my uh presentation and actually it’s closely related to christiana’s project um this U Museum x. uh platform so um when we developed this project uh out of the uh participatory processes uh we had in our
Museum um the the the main incentive was uh the co pandemic um that we thought okay we need something where we can react immediately to a situation like that and not with an exhibition two years later or something so uh we thought about an a platform and one of
The um first ideas to involve our curators too was um uh we we put up an um a platform organized like a social media platform where everybody could post or um put in objects uh stories Impressions um experiences that are connected to this pandemic uh so uh we
Could bring it together of in our on our platform um and pick up people where they are on Instagram Facebook uh uh whatever platform they they are using they could um put on a hashtag and recollect all this material on our um platform that was one idea we were
Pursuing there they came up a lot of projects that were collecting the pandemic then later um and we we we uh we abandoned this idea and came up with the more general idea to develop a platform uh that wouldn’t be so um so connected to the pandemic but could be
Used for all kinds of um uh rapid responses dialogue processes uh uh participatory collection um uh projects and so on um and I think uh this is uh an important perspective um and it will become even more important in in in the future and there comes the second Point
Um um of our approach in uh to use artificial intelligence because we we want to use artificial intelligence not for um classical curation processes there are a lot of um projects going on to to help curators uh to to better curate their collections but the the approach
We are pursuing is we want to put this these uh AI systems in the service of our users so they can curate themselves uh so you put it together you have a platform and you have a nice set of AI tools you can give your users and they
Have um a good um um nice and accessible um um user designs uh they could do a lot on our platform um with new collections uh Al also they could um create their own exhibitions on on a platform like that we are we are we are thinking and discussing this a lot at
The moment maybe that much um um Elena do you also want to um Sandra can ask a question okay yes um first of all I’d like to thank you Yan and Elena for two wonderful presentations I really enjoy them both and my question is for you Elena
Um if I’m not mistaken AI is a system which learns by doing the the more it it’s it’s it’s um it moves forward in time the more it becomes knowledgeable the more it can be accurate so I can see a risk there in the sense that if AI is being applied or
Is being adopted by a museum in Germany and AI is being adopted by a museum in the Mediterranean there are different languages coming together so on the other hand however museums are working with with standardization so we we’ve all been working with the with the with the with
The gy system and going for for for um a very specific set of of um of categorization which from you’re saying is now at risk that might not be functional that might not be I mean there might be a mismatch so how do you see this very
Specific um application of AI and the language that is in a way in a way cultural is being reflected in the culture of where that the AI is being applied and the need to have a new set of Standards a new a new way of of um cataloging
Okay so um in terms of um different museums of different countries using it I think there needs to be um yeah that collaboration because you know U museums have um similar um like eras cover in their collections so I think mums to be able to use and improve the results
Effectively there need to be that collaboration you know if you have an Egyptian collection you know at the med with one at the British museum with the one at you know other museums uh in in the world um and I will help to train the algorithm and make it better one
Thing that I know uh is that I know Wikipedia Wikipedia they are working on as well uh with AI and a way of doing that and I know the met and other museums are working on that is how you link that data as well we Wiki data
Records and Wiki data then is translated so there is a way of um mean I’m simplifying it here the the process is more complex than that but there there there there is a huge potential of having those TXS and that data and then translate to multiple
Languages um and the other the part the second part was about yes you were you are mentioning this this this potential mismatch or the risk that the catalog isation the the standards are becoming obsolete or in 10 years they will be obsolete because there the new standards
It’s what is what johanes was was mentioning for example know this this this new way of looking at objects and these new tags this this new way of yeah I don’t think that it that cataloging system will be uh obsolete still like there so much like thought and like work behind
That and those like gy vocabularies and how museums are cataloging their records however I think and and based on the case studies presenting presented on the AI project was that computer vision can bring like new ways you know it could be a comp ment a way of complementing the
Museum data so you’re not going to attack like colors you know in in in your you know manually but computer vision can do that very easily and with a good accuracy or maybe you know what is depicted on on our work it can help for uh those uh those things and then I
Mean the next thing is what johanes also mentioned is like okay you have a cators like language you have the computer vision language and another thing that I’m very interested is in okay what are museums what are Museum visitors actually searching on Google and on the Museum’s
Website to find um because that language is very different and I don’t know if you were familiar with the social tagging projects that were taking I can’t remember like maybe seven 10 years ago years when you invited the crowd you know to the crowd source that um process
I think was um Brooklyn Museum the PowerHouse Museum so those project showed again different ways people um describe the collection Johannes may I add maybe one point um um so uh there’s no misunderstanding I think it is still absolutely necessary to have the scientific cataloging of Collections and so as uh foundation of
The whole process but I think um in the future uh um and it’s in in my view already visible uh it will be a large part of the work of museums um to communicate between data sets of your content and uh the data of your users uh so um I think we
Are at the moment um in a process uh we should start to think about uh user data strategies yeah because the discussion about digitization and digitality in the museum sector was for a long time about content digitization but we we need also strategies for the user data yeah and I
Think one of the most interesting um uh points of entry for AI uh um applications is this communication between Museum data and user data uh so uh uh and you can take your your own example for ex language yeah you you already have um AI driven platforms like
L um and several others that produce astounding results in in in translating yeah that’s not categorization but translating um and some museums are starting to use these uh tools uh to automatically translate stuff um and if you if you take the usual situation in museums that you have two or three
Languages maybe for the description of objects if if at all yeah maybe you have only one language German English or whatever or a language is used by curators maybe for for several decades or so uh you can use these these approaches to make the these data accessible yeah you you can give users
Automat uh automatic uh translations that may not be perfect but um at least there is something yeah um to to to to get access to the collection okay um is there we’re approaching fast uh the end of our session um is is there one last question maybe that um is one burning question
Left because yeah well it’s it’s too little time for this topic but maybe I I just throw it in I’m always wondering uh what um because Elena you mentioned uh biases and you know data sets with AI what what the potential is because we we focused a lot on diversifying the
Collections in the first two sessions with quering and also with diversity in general is do you see any potential apart from the bias fears uh and and dangers for digital tools and AI to Foster diversity in in in collection uh collecting processes um so challenging um I’m sorry
Um I mean there’s so lack of diversity and that’s one of the things we talk about in the museum sector in general in how museums have been collected and I think what um the analysis of the collection data showing is that lack of diversity and how that already will
Create a bias when you train the algorithm um yeah I mean I think that cancer face like the current um lack of diversity issues of Museum Collections and can bring like new uh policies about collecting um but um I mean it’s very I don’t know I don’t have the definitive
Answer about it or is it’s I would could spend like another like hour and a half just on diversity in collections and also on these algorithms the way have been trained in many cases um is also based on already some biased data like image net is probably the most obvious case
That can come to our minds um and yeah I I don’t have the definite answer about how to solve that issue but um I think caner face some issues about lack of um of diversity thank you um um yeah is there any anything else that I think we could discuss
Forever I I have the feeling because there’s so many topics that were mentioned and uh also in already in the um in the in the talks um but I was talking to danela of maybe we we we do a um um maybe later in the year uh like a
Uh a second session or a check-in session of what uh if if there’s uh um if you want to discuss more but uh I think we heard a lot today about uh ways to to uh about participation and about Ai and also problems and challenges but
Also great potential um I want to thank my two guests Elena and Johannes for their insightful presentations and and really all of you for joining this discussions I I really enjoyed it and I think really yeah we could talk for forever I think um we um uh we’re going to uh um post
The recording of the session uh soon on the we are museums Channel and also uh in one uh one next month on February the 18th the next session of reframing collection collecting would be on collecting memories so if if you if you’re also interested in that please
Check in and join the the we museums Community there’s a lot of interesting uh talks uh happening and for now I it’s just uh yeah left to to say thank you again for this nice start of the afternoon in my in my case and uh one
And a half hours of talking thank you very much thank you everyone yeah thanks bye thank you bye bye you very interesting