Ddgr doesn´t work anymore

Hi all, :wave:

the weirdest thing happened just now.

Every now and then I want to use ddgr (for data saving…) and suddenly it doesn´t seem to work anymore. :slightly_frowning_face:

ddgr is a cmdline utility to search DuckDuckGo (html version) from the terminal.

I can´t tell exactly when I used it last ( perhaps 2 weeks ago) but back then it worked.

ddgr can be found here: GitHub - jarun/ddgr: DuckDuckGo from the terminal .

I use the standalone binary as I don´t want to have it installed:

Running standalone

ddgr is a standalone executable (and can run even on environments like Termux). From the containing directory:

$ ./ddgr

The command I use is:

cd /media/rosika/f14a27c2-0b49-4607-94ea-2e56bbf76fe1/DATEN-PARTITION/Dokumente/work/ddgr-1.9/ddgr-1.9; and env BROWSER=w3m firejail --private=(pwd) ./ddgr; and cd

(fish-syntax)

and normally it works beautifully. :wink:

I assume it has something to do with the latest kernel but I can´t be definite about that of course. :thinking:

The latest kernel on my system is: linux-generic:amd64 5.4.0.126.127 and I installed it on 2022-09-20.
So I guess the last time I used ddgr was before that date.

To exclude the firejail sandbox from being the culprit I issued the command
./ddgr
from the respective folder. I still got the same error message. :slightly_frowning_face:

Now to the error itself:

ddgr started normally at first:

ddgr (? for help)

When I entered a search term (e.g. “bbc”) it just said:

[ERROR] HTTP Error 403: Forbidden
No results.

The same happens with any search term. :angry:

Does anybody of you make use of ddgr and do you experience this behaviour as well :question:
Any idea what this might be all about (all of a sudden) :question:

Many thanks in advance and many greetings
Rosika :slightly_smiling_face:

BTW:

I also got hold of the latest version (2.0) and tried it with that one as well. Still the same result.

1 Like

Hi Rosika,
Excuse me getting off topic, but I do not understand .
Do you mean … without having it installed by the package system?
So how did you get the binary? Did you just download the source code tarball?

Now back on topic
When I looked at your site, ddgr on Github, it says it depends on certain things (Eg python3). Is it possible that one of the dependencies has been upgraded in your system.
One way to check would be to try it in you Debian in VM. That would probably not be upgraded.

This may not be relevant, but I am having the same sort of issue with curl at the moment. It simply refuses to do even the most elementary download… I had to go back to wget. I dont suppose there has been any change in internet behaviour?

Regards
Neville

1 Like

Thanks, Neville, for your reply, :heart:

Yes, you can run ddgr perfectly from the standalone-binary.

The direct download of the latest version is:
https://codeload.github.com/jarun/ddgr/zip/refs/tags/v2.0 .

Or: Release ddgr v2.0 · jarun/ddgr · GitHub . Just choose “Source code (zip)” and then unpack it to a dedicated directory.
From there you just run
./ddgr

Good thinking, Neville. Thanks a lot . I guess that should be it.

E.g. my latest python update was on 2022-08-31. Could be the culprit… :thinking:

Great idea. :+1: Thanks.

I´ve done it and indeed: here it works :smiley:

Example with search-word “bbc”:

rosika2@debian ~> cd /home/rosika2/Dokumente/Ergänzungen_zu_Programmen/zu_ddgr/ddgr-2.0; and env BROWSER=w3m firejail --private=(pwd) ./ddgr; and cd

ddgr (? for help) bbc

     The British Broadcasting Corporation is the national broadcaster of the United Kingdom. Headquartered at Broadcasting House in London, it is the world's
     oldest national broadcaster, and the largest broadcaster in the world by number of employees, employing over 22,000 staff in total, of whom approximately
     19,000 are in public-sector broadcasting. The BBC is established under a royal charter and operates under its agreement with the secretary of state for
     digital, culture, media and sport.                          More at Wikipedia

 1.  BBC - Homepage [www.bbc.com]
     The most amazing videos from the BBC, The supernatural powers of the African Joan of Arc, Two young women's thoughts on having an 'intense and fulfilling
     love affair', A surprising South Korean...

 2.  BBC - Home [www.bbc.co.uk]
     The most amazing videos from the BBC, The supernatural powers of the African Joan of Arc, Two young women's thoughts on having an 'intense and fulfilling
     love affair', A surprising South Korean...
[...]

Hmm, :thinking: I guess there´s nothing much I can do about it then…

Thank you so much, Neville, for your kind help.

Many greetings from Rosika :slightly_smiling_face:

Well that proves it… it is a matter of versions of python3, or some other dependency

What you need is ddgr in a docker container where you can provide the right version of python etc, without mucking up your Lubuntu.
That would not be hard. If you want to try and make a Dockerfile, I can help.

Regards
Neville

1 Like

Thanks so much Neville,

Thanks a lot for your kind offer :heart:.

As a matter of fact I haven´t installed docker on my main system (Lubuntu) yet, just on my Debian vm in order to learn… :blush:.

So that would be a separate project. I´d like to defer it for a while. But it´s great to know you would help. Thanks so much. :heart:

Well, as a workaround I can still use w3m and fire up duckduckgo.com from there.
In fact I alraedy bookmarked the page.

BTW:

As ddgr is also available as a deb-package I issued the command
env LANG=en_US.UTF-8 apt-get -s install ddgr
to see what would be installed:

env LANG=en_US.UTF-8 apt-get -s install ddgr
NOTE: This is only a simulation!
      apt-get needs root privileges for real execution.
      Keep also in mind that locking is deactivated,
      so don't depend on the relevance to the real current situation!
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  ddgr
0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
Inst ddgr (1.7+git20190928.bccdc92-2 Ubuntu:20.04/focal [all])
Conf ddgr (1.7+git20190928.bccdc92-2 Ubuntu:20.04/focal [all])

It´s the 4 not upgraded remark that makes me think… :thinking:

Many greetings
Rosika :slightly_smiling_face:

They must be dependencies of ddgr.
apt-cache show ddgr should list them

I think it means ddgr wanted newer versions of 4 packages, but apt would not allow the upgrades, presumably because other things needed the current versions and may break if the upgrades took place.

Good thinking, looking at that .
Regards
Neville

1 Like

Thanks, Neville.

Ah, thanks. I looked it up:

 env LANG=en_US.UTF-8 apt-cache show ddgr
Package: ddgr
Architecture: all
Version: 1.7+git20190928.bccdc92-2
Priority: optional
Section: universe/misc
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: SZ Lin (林上智) <szlin@debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 111
Depends: python3:any, python3 (>= 3.5)
Filename: pool/universe/d/ddgr/ddgr_1.7+git20190928.bccdc92-2_all.deb
Size: 32048
MD5sum: d0c8a82d8fe2b2d7699bb2fdf6c06d02
SHA1: 6a91495a477e7eefc64d6053cf8ceb5240a0d1a9
SHA256: 482c1750476c83c5a00c5516d5e1f122afca9d6294989271d6fb217f0aef7312
Homepage: https://github.com/jarun/ddgr
Description-en: DuckDuckGo from the terminal
 Features
 .
  - Fast and clean (no ads, stray URLs or clutter), custom color
  - Navigate result pages from omniprompt, open URLs in browser
  - Search and option completion scripts for Bash, Zsh and Fish
  - DuckDuckGo Bang support (along with completion)
  - Open the first result directly in browser (as in I'm Feeling Ducky)
  - Non-stop searches: fire new searches at omniprompt without exiting
  - Keywords (e.g. filetype:mime, site:somesite.com) support
  - Specify region, disable safe search
  - HTTPS proxy support, Do Not Track set, optionally disable User Agent
  - Support custom url handler script or cmdline utility
  - Comprehensive documentation, man page with handy usage examples
  - Minimal dependencies
Description-md5: fbf97be4352d256c7db3959fdb002831

As dependencies it lists just python:

Depends: python3:any, python3 (>= 3.5)

Hmm, my installed version of python3 seems to meet the demands: :thinking:

python3 --version
Python 3.8.10

Thanks a lot, Neville.
Regards
Rosika :slightly_smiling_face:

Hi Rosika,
It does. End of that theory.
So we do not understand what the 4 packages were and how they are relevant to ddgr?
I’ll sleep on thst
Regards
Neville

1 Like

Thanks, Neville, for the confirmation.

Right, that puzzles me too…

In the meantime I also put the issue forward to the github-forum under “issues”.
Being the only poster there at the moment I wonder if I might get an answer there. :thinking:

ddgr doesn´t work anymore · Issue #147 · jarun/ddgr · GitHub .

Thanks a lot, Neville.

Many greetings from Rosika :slightly_smiling_face:

1 Like

SOLUTION: :smiley:

I found the solution.

In actual fact the problem was already discussed some days ago but the issue was marked as solved and not visible at first.

Anyhow: error: 403 (forbidden) · Issue #146 · jarun/ddgr · GitHub led me to the sloution:

I have a workaround.
I changed the user-agent in ddgr to:

USER_AGENT = ‘Mozilla / 5.0 (X11; Linux x86_64; rv: 104.0) Gecko / 20100101 Firefox / 104.0’

Now it is working.

(only sometimes i get:)
[ERROR] <urlopen error [Errno 113] No route to host>
No results.

So I looked it up on: Fix #145 #146: change user agent · jarun/ddgr@02025e8 · GitHub .
Here is the critical part:

in the file ddgr

file ddgr
ddgr: Python script, UTF-8 Unicode text executable

we have to change line 88:

USER_AGENT = 'ddgr/{} (textmode; Linux x86_64; 1024x768)'.format(_VERSION_)

to

USER_AGENT = 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0'

Indeed USER_AGENT has to be changed to get it working again.
I tried it and it really works now :+1: .

I still haven´t got any clue why this change was neccessary, i.e. something must have changed after all.
But I´m glad this could be solved.

Thanks @nevj for you kind help. :heart:

Many greetings
Rosika

2 Likes

I dont understand it either, but I do admire your diagnosis.
Ddgr works in Debian, and does not require USER_AGENT changes, so Debian bust have an older version of ddgr?

Sometimes you can get a better answer by looking at or posting to Github Issues on a relevent repository , than by searching websites.
I have done that a couple of times with Void Linux, and it was helpful.

Regards,
Neville

1 Like

I just tried it out as an “installed app” on Ubuntu 22.04 (apt install ddgr) and thought - where has this been hiding all my life! It’s great!

So - I saw the “termux” bit in the blurb, but - my TermUX is hosed - thanks to Google - half of the stuff (and the repos) are broken on Google’s App Stores (seems to be happening an AWFUL LOT lately! And I can’t install ddgr… :sob: This is really starting to cheese me off…

The ONLY way to reliably run TermUX these days (and I PAID for the STUFFING thing!) is to “side load” (what a SHIT terminology) it using “FDroid” - and - I can’t install FDroid 'cause my Galaxy S9+ is NOT rooted (in Australian and NZ English - “rooted” basically means “f–ked”) - and - I can’t root it 'cause I need it for work and I work are very strict - if I want to use their Teams and Outlook - I have to play by certain rules - and - I don’t know if the MFA apps I NEED (there’s at least FOUR of them I need!) for my job will work a rooted Android or via FDroid …

1 Like

Hi Neville, :wave:

Thanks for your compliment. :heart:

To be fair, I didn´t do much of a diagnosis myself but followed the workaround instructions
given on the github pages (see: the two links in post #10).
I quoted the relevant passage… didn´t want to pass it off as my “invention” of course. :blush:
I tried it and it worked.

Right. That I did. But thanks for the reminder. :+1:

Actually no. I just copied the same executable binary I use on Lubuntu over to my Debian vm (via ssh).

BTW:

Thunar file manager (Nautilus as well) natively supports the SSH protocol.
To do this, use an address in the form ssh://computer name/path to see the files on the specified computer via SSH.
If you want to log in as a different user, use an address of the form ssh://otheruser@computername/path instead.

(translated from German from SSH › Wiki › ubuntuusers.de )

Once the Debian vm is booted by
LIBVIRT_DEFAULT_URI=qemu:///system virsh start debian10
I can easily log into it via thunar and transfer files and folders. :blush:

(Sorry for the digression topic-wise)

So I really did use the same version of ddgr on both systems.

Many greetings
Rosika :slightly_smiling_face:

Thanks, Dan, for your comments.

I´m pleased to hear you like it. :+1:

I must admit I hadn´t heard of termux before so I had to look it up:

Termux is an Android terminal emulator and Linux environment app that works directly with no rooting or setup required. A minimal base system is installed automatically - additional packages are available using the APT package manager.

(Termux | The main termux site and help pages. )

Well, from what you said I gather you´re in a bit of a dilemma there. Sorry to hear that. :slightly_frowning_face:
As I have no experience with things like that I fear I cannot be of much help there…

Still: thanks a lot and many greetings.
Rosika :slightly_smiling_face:

Oh, then why it works in Debian has to be that its dependencies are different versions in Debian.
Its not worth the trouble to check that. Lets move on.

So ssh works to a computer inside a VM.
Must try it with Vbox… I doubt if if would work there. Also would it work into a docker container? A container can see files on the host so ssh would be unnecessary.
I wonder if rsync would work the same way? It uses ssh.

Regards
Neville

1 Like

You´re certainly right there. Thanks.

Yes. In order to log into a running vm I do the following:

1.) this isn´t necessary; just to give an overview of what´s available :wink:

LIBVIRT_DEFAULT_URI=qemu:///system virsh list --all
 Id   Name              State
----------------------------------
 -    archlinux         shut off
 -    clonezilla        shut off
 -    debian10          shut off
 -    kgw_versuch_hük   shut off
 -    ubuntu18.04       shut off
 -    vm1               shut off
 -    vm2               shut off

2.) LIBVIRT_DEFAULT_URI=qemu:///system virsh start debian10
… starts Debian vm

3.)

LIBVIRT_DEFAULT_URI=qemu:///system virsh domifaddr debian10
Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet0      52:54:00:21:8b:6a    ipv4         192.168.122.174/24

… shows Debian´s IP address

4.) firejail ssh rosika2@192.168.122.174

ssh-ing into the running vm (even sandboxed :wink:)

The last step is not necessary if I just want to transfer files. In this case I do it via the aforementioned method with thunar:

All this refers to qemu/kvm… :wink:

Hmm. I don´t want to say anything that might me wrong.
Off the top of my head I can´t tell. I´d have to investigate…
In the meantime: perhaps someone else might know the answer… :thinking:

Thanks a lot, Neville.
Many greetings from Rosika :slightly_smiling_face:

I get it, you have to start Debian in the VM, before you can ssh into it.
I dont think Vbox has those sort of external commands. It is all graphic.

You are convincing me I should learn qemu. It is much more flexible.
Thanks for showing me that

Regards
Neville

1 Like

Hi Neville, :wave:

you´re welcome. :heart:

Yes, the vm has to be running for that.

Years ago I was using virtualbox as well but wasn´t particularly satisfied with it.
Purely subjective of course and I even don´t know the exact reasons any more.
After that I switched over to vmware which suited me better.

But when I learned about qemu/kvm and virt-manager I was pretty convinced of that solution.
Highly configurable, I guess and very well documented.

If installed on your system you get a huge help-file locally on /usr/share/doc/qemu/qemu-doc.html by default and online resources are available too, of course.

:wink:

Many greetings from Rosika :slightly_smiling_face:

1 Like

I am finding rhe same. Vbox is a beast to set up.
Gnome Boxes is easy, but very limited.
I have a friend who uses VMware. He is happy with it, but he does not ask much of it.
I think a move to qemu is my best option. From what you have been doing it is much more flexible. I just needed a start, and you have given me enough examples for that . Thank you…

Regards
Neville

1 Like

OK - not sure (pretty sure I’m not adding any value - apologies - other than confirming I can get similar symptoms) this is helpful - but :

  1. it works flawlessly and I’m impressed, and will use it more often, installed it (via apt) in Ubuntu - this behaviour “works out of the box” - e.g. :
╭─x@titan ~  
╰─➤  ddgr 1986 stand by me
  1.  Stand by Me (1986) - IMDb [www.imdb.com]
     Stand by Me: Directed by Rob Reiner. With Wil Wheaton, River Phoenix, Corey Feldman, Jerry O'Connell. After the death of one of his
     friends, a writer recounts a childhood journey with his friends to find the body of a missing boy.

...
 10. Stand by Me (Ben E. King song) - Wikipedia [en.wikipedia.org]
     "Stand by Me" (1986) "Spanish Harlem" (1987) "Stand by Me" is a song originally performed in 1961 by American singer-songwriter Ben E.
     King and written by him, along with Jerry Leiber and Mike Stoller, who together used the pseudonym Elmo Glick. According to King, ...

But - not sure what I’m doing wrong on Debian (11 - buster running on a Pi4) - ddgr installed by the same method : “apt install ddgr” :

╭─x@beere253 ~  
╰─➤  ddgr -d 1986 stand by me
[DEBUG] ddgr version 1.6 Python version 3.7.3
[DEBUG] q:1986 stand by me, region:us-en, page:0, curindex:1, safe:1
[DEBUG] nextParams:
[DEBUG] proxy:{'https': None}
[DEBUG] ua:ddgr/1.6 (textmode; Linux x86_64; 1024x768)

[DEBUG] Response body written to '/tmp/ddgr-response-shh_9xwx'.
[DEBUG] Prev nextParams: 
[DEBUG] Next nextParams: 
No results.
ddgr (? for help) 
ddgr (? for help) 1986 "stand by me"
[DEBUG] q:1986 "stand by me", region:us-en, page:0, curindex:1, safe:1
[DEBUG] nextParams:
[DEBUG] proxy:{'https': None}
[DEBUG] ua:ddgr/1.6 (textmode; Linux x86_64; 1024x768)
[DEBUG] Response body written to '/tmp/ddgr-response-tqvoioem'.
[DEBUG] Prev nextParams: 
[DEBUG] Next nextParams: 
No results.
ddgr (? for help)

Now - I took a shortcut - because despite that URL mentioning TermUX - it won’t install (after spending the afternoon fixing up the spaghetti mess my TermUX apt sources were in - thanks to the vendor and Google) - it can’t install it - i.e. “pkg install ddgr” on TermUX says pretty much: “bugger off you fool that doesn’t exist anywhere!”

My shortcut - TermUX when install properly can run sshd, simple as “sshd” - and I can SSH to it (I don’t run it on port 22!!!) - so I scp’d the aarch64 binary from my Pi4 to my Samsung phone (I call it “samdung”) :

╭─x@beere253 ~  
╰─➤  scp /usr/bin/ddgr samdung9.local:~/.
ddgr                                                                                                           100%   65KB   2.0MB/s   00:00

SSH to my phone :

╭─u0_a206@localhost ~  
╰─➤  ./ddgr monteverdi orfeo
No results.
ddgr (? for help) ?

omniprompt keys:
  n, p, f               fetch the next, prev or first set of search results
  index                 open the result corresponding to index in browser
  o [index|range|a ...] open space-separated result indices, ranges or all
  O [index|range|a ...] like key 'o', but try to open in a GUI browser
  d keywords            new DDG search for 'keywords' with original options
                        should be used to search omniprompt keys and indices
  x                     toggle url expansion
  c index               copy url to clipboard
  q, ^D, double Enter   exit ddgr
  ?                     show omniprompt help
  *                     other inputs are considered as new search keywords

ddgr (? for help) monteverdi orfeo
No results.
╭─u0_a206@localhost ~  
╰─➤  file ddgr
ddgr: Python script, Unicode text, UTF-8 text executable

Which seems to indicate that ddgr isn’t a binary - its a python script… So I copy the one from my ubuntu x86_64 machine to the Samsung piece of crap :

╭─u0_a206@localhost ~  
╰─➤  ./ddgr-ubuntu monteverdi orfeo 
[ERROR] HTTP Error 403: Forbidden
No results.
ddgr (? for help)

Now I’m guessing I’m probably doing something wrong, maybe I need to configure something - but it’s annoying it works out of the box on Ubuntu 22.04 and 22.10 without ANY configuration (ubuntu 22.10 beta) :

x@fenriz:~$ ddgr monteverdi orfeo

 1.  Monteverdi's Orfeo: 'a brilliant and compelling fable to the ... [www.theguardian.com]
     Monteverdi might be surprised to find himself hailed as the inventor of the opera, and he disclaimed the role of revolutionary, but his Orfeo is a radical,
     innovative and extraordinary work that ...
...
 10. Claudio Monteverdi: Orfeo - Todd Tarantino, composer [toddtarantino.com]
     Claudio Monteverdi's Orfeo is widely considered the first great opera. The libretto, penned by Monteverdi's friend Alessandro Striggio, relates the legend of
     Orpheus and his descent into the underworld to recover his beloved, Euridice. Monteverdi's opera was commissioned by a wealthy group of Mantuan nobility known as
     the Accademia degl ...

ddgr (? for help)

I prefer my food to come cooked already, I prefer my CLI software to work out of the box without tweaking and configurations :smiley: and I prefer my opera to have a fat lady singing at the end, although in the case of Monteverdi’s D’Orfeo, I’ll make an exception :smiley: - in fact I vastly prefer Italian Baroque opera to all that German stuff by Wagner and Strauss…


So - I can confirm - all the places / use-cases / situations, where I might care to use ddgr, don’t work… I have no need for it on Ubuntu whatosever! But it would be VERY hand to use on my Pi4 system, and via TermUX on my Samsung handset - and it doesn’t…

╭─u0_a206@localhost ~  
╰─➤  ./ddgr --version                                                                                                                                                 1 ↵
1.6
╭─u0_a206@localhost ~  
╰─➤  ./ddgr-ubuntu --version
1.9

That “ddgr-ubuntu” is just the python script I scp’d from my Ubuntu box - it’s python so don’t need to worry about aarch64 vs x86_64…
works on MacOs via brew too - but I don’t need it on full blown desktop systems with industry standard web browsers (note - it’s version 1.6 in buster, version 1.9 in Ubuntu, and 2.0 in homebrew on the Mac).


also confirm - it’s version 1.9 on Bullseye (raspbian) and it’s broken there too…

 x@albiorix  ~  ddgr monteverdi orfeo
[ERROR] HTTP Error 403: Forbidden
No results.
2 Likes