Here’s the toy problem - given those pictures of bookshelves, I’m trying to figure out what authors to reach out to. The biggest thing I’m looking for is evidence that the author has an actively engaged user base, but I also want to know more about the book and the author’s overall body of work. It would be pretty great to be able to go from a picture of a bookshelf to however many profiles of authors and their social media presence.
I did my trials with Emily Oster as the subject. She’s the author of “Expecting Better”, part of the ParentData series. I don’t think her work is a great fit for the kind of companion experiences I have in mind (and she’s already launched Dewey), but I’m fairly familiar with her as an author and her online presence so I can evaluate ChatGPT’s performance marginally better than I might otherwise.
Takeaways
“Browse with Bing” leverages web searches to return more up-to-date information. This is especially important if you need ChatGPT to take into account anything after Jan 2022.
Links were not always right, and it was hard to get ChatGPT to reflect on its answers and improve.
A lot of sites block OpenAI, so some basic tasks (“how many followers does Emily Oster have on Instagram”) were impossible.
“Browse with Bing” had a much higher error rate than vanilla ChatGPT in my experience.
Availability
I ran these queries last week. Although some reports indicated that “Browse with Bing” could be found in the chat.openai.com settings, I actually discovered it by hovering over the model toggle at the top of the page.. By Monday I’d lost access - online reports suggest that features like this roll out in stages, and apparently they’re sometimes rolled back.
This is where “Browse with Bing” had appeared.
Overall
I used the following two prompts:
For the following book, please return a brief summary, the date published, and the name of the author: "Expecting Better"
I read the book and really liked it. Does Emily Oster actively post related content online, maybe to social media, a website, an email newsletter, or a podcast? If so, please provide links to those sources.
For the first overview question, both web and non-web responded reasonably. The cool thing about the web response is that it responded with source links, which makes it a little easier to verify.
Here’s the non-web response for the second prompt:
As of my last training data in January 2022, Emily Oster is an active academic and writer who does share content online. Here's how you can follow her work:
Twitter: Emily Oster is fairly active on Twitter, where she comments on current events, shares her latest work, and discusses topics related to her areas of expertise. Her Twitter handle is @ProfEmilyOster.
Website: She has a personal website, emilyoster.net, where she posts articles, information about her books, and other resources.
Newsletter: Emily Oster has been known to write for various media outlets and has also started a newsletter on Substack called "ParentData". You can check it out and subscribe to it at [url]. [Rob: Substack automatically turns this link into a widget if I paste it bare]
Podcast: As of my last update, Emily Oster doesn't host her own podcast, but she has been a guest on several podcasts. You can find her appearances by searching her name in your preferred podcast app.
Remember, online content and platforms evolve, so it might be worth doing a fresh search to see if Emily Oster has expanded her online presence or added new platforms since 2022.
This is all pretty much right. Emily Oster does have a podcast, but the first episode was released on April 12, 2022, after ChatGPT’s latest snapshot. The most significant advantage of web-enabled LLMs is their ability to provide up-to-date information, so hopefully we see some improvement. Here goes (footnotes are links provided by ChatGPT):
Emily Oster actively engages with her audience online through various platforms:
Website: Emily Oster's official website is a central hub where she shares her work, writings, and research on topics related to pregnancy and parenting1.
Substack Newsletter: She runs a Substack newsletter titled "ParentData" where she discusses data, pregnancy, and parenting1.
Social Media:
TikTok: Emily Oster is active on TikTok where she shares content related to her field of expertise1.
Instagram: She also shares content on her Instagram profile1.
Twitter: Emily Oster can be found tweeting on Twitter about various topics including pregnancy and parenting1.
LinkedIn: Her LinkedIn profile is another platform where she maintains a professional presence1.
ParentData Organization: Emily Oster has also initiated the ParentData organization, which is mentioned on her website, likely as an extension of her mission to create a world of more relaxed pregnant women and parents1.
COVID-19 School Data Hub: Additionally, she launched the COVID-19 School Data Hub, which provides a detailed look at school operations during the 2020-2021 school year in relation to COVID-19 case rates2.
Through these platforms, Emily Oster continues to share her insights and data-driven approach to pregnancy and parenting, making information accessible to a wider audience.
This is better! It has more social links and more contemporary commentary. It still misses the podcast, and weirdly it has the wrong LinkedIn link. I found that pretty surprising because its source appears to be Oster’s website, which has direct (and correct) links at the bottom.
Links
I found links to be generally challenging for it. Here was my first “generate a profile on this author” request:
For the following book, please return a brief summary, the date published, an estimated number of copies sold, and the Amazon link for the book (if available). Please respond in JSON. "Expecting Better" by Emily Oster
Here is the information for "Expecting Better" by Emily Oster in JSON format:
The exact sales figures are not publicly available. Additionally, I wasn't able to retrieve the Amazon link for the book due to restrictions on accessing the Amazon page. However, you might find the book on Amazon by searching for its title and author.
I shouldn’t have asked for JSON, I was getting ahead of myself. Generally it seems like you should get your questions right before you go asking for formatting.
Not being able to get the Amazon link surprised me, but makes a kind of sense - lots of companies would prefer that OpenAI not ingest their pages.
Is there another link that you might recommend instead of the Amazon link?
Certainly! You might consider purchasing "Expecting Better" by Emily Oster from Penguin Random House's official website. Here's the link to the book on their site: Expecting Better on Penguin Random House.
Great! Except that url resolves to “The Asylum” by Simon Doonan, a completely unrelated book. Here’s the URL
Penguin’s site apparently doesn’t care about the text in the url, it’s just looking for that numeric ID. The correct link is the first organic hit on Google and Bing for “penguin random house expecting better”, and after a search for the wrong id I can’t find where it’s getting “312570” from. I had ChatGPT regenerate its response, but got the same wrong link back.
I tried to get it to evaluate that link a little more closely:
When you follow that link, what HTTP code is returned?
I was able to follow the link successfully, but the HTTP status code was not visible from the content displayed. However, the page loaded properly to a different book's page, indicating a likely successful response such as HTTP 200 OK. If you are looking for a technical verification of the HTTP status code, you might consider using tools or browser extensions like HTTP Status Checker or inspecting the network traffic using your browser's developer tools.
It can’t read the HTTP status codes, correctly identifies that the page loads to a different book, and then incorrectly infers a 200 (it returns a 302 “Temporary Redirect”).
I was redirected unexpectedly to this page: https://www.penguinrandomhouse.com/books/312570/the-asylum-by-simon-doonan/
It seems like the link to "Expecting Better" redirected to a different book's page. This might be due to a change on Penguin Random House's website or an incorrect URL. You might want to try searching for "Expecting Better" by Emily Oster directly on Penguin Random House's website or on other online book retailers.
It seemingly can’t try again, or use a different path when confronted with the failure of the first. I was really surprised that it prompts the user to search on their own. This suggests to me that step-by-step prompt engineering (where you’re asking it to show its work or check its work after the fact) might not work right out of the box for problems like this.
My guess is that it’s making its requests purely against a cache or index, which isn’t the architecture that immediately comes to mind. I could imagine it using your browser and credentials to make requests in the future, such that it’s returning web data that’s available to you rather than available to it.
Follower Counts
What would really help in terms of triaging who I should reach out to is working up profiles that include follower counts. This should be a pretty easy job - click through each social link, then return the number of followers. I updated my second prompt to the following:
I read the book and really liked it. Does the author actively post related content online, maybe to social media, a website, an email newsletter, or a podcast? If so, please provide links to those sources, and where possible list their follower count.
This response is slightly edited to remove some incorrect links.
Emily Oster actively shares content related to pregnancy, parenting, and data analysis across various platforms. Here's a summary of her online presence along with the associated links:
Website: emilyoster.net - Contains links to her social media profiles and other online platforms7.
Contact: You can contact Emily Oster for speaking engagements or other inquiries through parentdata.org8.
You may visit the provided links to follow Emily Oster on these platforms and stay updated with her content.
The only follower count provided is TikTok - but both the TikTok and Instagram accounts linked are for other people. The links it provided on the first try were correct. For what it’s worth, the TikTok follower count was correct (for that incorrect account).
It seems like ChatGPT is restricted for nearly all the sites I’m looking for, but maybe not Substack:
How many subscribers does Emily Oster have on Substack?
Emily Oster's exact subscriber count on Substack is not publicly disclosed on her Substack page or other sources. However, it's mentioned in multiple places that her Substack publication, ParentData, has "hundreds of thousands of subscribers"12, and among them, there are "tens of thousands of paid subscribers"3. For the most accurate and updated subscriber count, it might be beneficial to reach out to Emily Oster directly through her contact page on parentdata.org or her official website4.
Here’s the first thing you see if you go to Emily Oster’s Substack:
Wrap Up
ChatGPT with web browsing shows promise - if your request needs information after Jan 2022, this feels like an important thing to try. However, even though my explorations aren’t particularly scientific, it seems clear that you’ve got to be even more skeptical of the web-enhanced responses.
