Jamf Pro 10 not mass deleting computers

ChupSuy
New Contributor III

I'm just testing the Jamf Pro 10 release in the lab and encounter a problem when deleting multiple computers @ once.

to make things a little more visible I enabled SQL statement logging on JSS.

I intend to delete 10 machines @ once by clicking Action --> Delete Computers.

After pressing "Next" on the choose an Action screen all I get is a waiting circle and after some time the Session is timing out. I do see the CPU for "mysqld" increasing but never recovering how long I even wait, nor does my computer count is decreasing by 10.

Also in the JSS log I can not find a single delete statement showing up..

Deleting Computers 1 by one seems to work. Did anyone else encounter this problem ?

36 REPLIES 36

dsweigart
New Contributor III

I noticed this too when I was attempting to delete more then 2 devices at a time. I ended up just “clicking” on my page every few minutes during the loading screen process and after about 10-15 minutes, I was asked to confirm the deletion.

I tried 50 devices and it kept timing out on me.

pnbahry
New Contributor III

I have also noticed this problem, I will need to remove over 200 machine records and it just times out and mysql is maxing out on CPU usage.

neil_martin83
Contributor II

Same here.

ChupSuy
New Contributor III

for now as a workaround I have made myself an API script feeding a txt file with Computer names and delete 1 by one via API instead but this must be fixed in the GUI as we have folks out there that frequently purge a large number of machines @ once which will then cause the DB to go down.. :/

I informed support and hope they'll open a PI on that..

chris_kemp
Contributor III

Yeah, same here - trying to clean up our dev JSS and it's not letting me mass-delete computers.

arpierson
New Contributor III

Seeing the same thing here. fwiw, I upgraded our MySQL to 5.7 today as well. So, I'm running the recommended config according to the Jamf Pro 10 documentation.

chris_kemp
Contributor III

I found that by bouncing tomcat then mysql recovers. It seems that other things kept functioning, at least to a degree - when I checked our db server it was running at over 400% cpu, which may be due to my having run more than one delete. Soon as I restarted tomcat it went back to normal - and executing another mass delete sent it right back up to around 130% usage, until I bounced tomcat again.

chris_kemp
Contributor III

Incidentally, we're running RHEL6 and mysql 5.6.

ChupSuy
New Contributor III

by enabling the Statement Debugging I found whenever you intend to delete more than 1 machines its doing insane a lot Select statement.. not sure what these shall be good for as all thats really needed for deleting the machines is the ID's..

That seems to be taking to long and the session is timing out before you even get to the screen Do you really want to delete these machines..

send my logs to the support..

jkuo
Contributor

I have not yet tested this on Jamf 10, but I've posted a command line API tool that will delete computer records reading from a CSV file one at a time. Since you're testing in a lab environment, perhaps this might work and save you some time?
https://github.com/eventbrite/Casper-API-Tools

And here's how the command would work:

https://github.com/eventbrite/Casper-API-Tools#delete-computers-by-jss-id-using-a-csv-file

Or you could just delete one at a time and build all the commands in a spreadsheet, but it would at least save you the time of going to each record. It would be a command like:

jamfapi deletecomputerbyid 1234

Where '1234' is the JSS ID of the computer.

Chris_Hafner
Valued Contributor II

On a test server (and old mac mini) I just deleted 653 computer records without issue. It took some time but it did. Configuration as follows: JSS 10 on macOS 10.12.6 "Server", MySQL 5.7. something... it crashed when I tried to log in to check the exact version. I guess we may still want to avoid 5.7 in favor of 5.6. Yet, this IS a test server so I find it interesting that it did manage to delete all those computer records and then muck itself.

jwojda
Valued Contributor II

I had it delete 25 users and it went through fine, a little slow, but didn't error out.

ChupSuy
New Contributor III

done some further digging on this and it turns out it all has todo with the historical data a given machine carries around..

Checking my DB tables I found my tables location_history & locations where insane large...

use jamfsoftware; SELECT table_name AS "Tables", Round(( ( data_length + index_length ) / 1024 / 1024 ), 2) "Size in MB" FROM information_schema.tables WHERE table_schema = "jamfsoftware" ORDER BY ( data_length + index_length );

So how to flush these keeping the most recent data only.. This answer I found on another discussion: https://www.jamf.com/jamf-nation/discussions/17908/user-and-location-history-flush-maintenance

Thanks @brunerd who shared his

Journey of a thousand commands

After I had cleared the historical data only keeping the most recent location history suddenly I could delete large counts of machines again..

Wish Jamf could provide a option to regularly flush these historical data tables on demand or on a schedule who we would not have to use direct DB manipulating..

Hope this helps anyone having similar issues..

arpierson
New Contributor III

@ChupSuy, this was the answer! I had 11.9 million rows in location_history and couldn't get anything to delete. Ran the commands on the linked page, got down to approx. 4700 rows, and deleted over 120 machines with no problems!

rcorbin
Contributor II

We were told there is actually a product issue open for this behavior. PI-004957.

In the mean time I may have to give that flushing historical data a try. Or the API script.

rcorbin
Contributor II

Has anyone tested @jkuo 's tool in Jamf 10 ? After clearing out tables location_history & locations it is better. I can do 20 at a time and it works. But If I try say 200 it eventually times out.

jkuo
Contributor

@rcorbin I've been using it in Jamf 10, but I have not done large batches, only smaller ones (maybe 5 or so at a time).

Br3ck
New Contributor III

@jkuo I just had a go with your deletecomputeridsfromcsv and it worked mostly great but you still have to confirm every deletion with a "y" and return but this is far better than trying to use the JSS web interface to delete one by one. I really cant believe this PI is still not fixed, its a big pain but this CLI was a lifesaver so thank you!

Also wanted to note that the action failed on a few of the objects I was attempting to delete:

Are you sure you want to delete the computer above from the JSS? (y/n): y
Deleting computer 2652...
Successfully deleted computer 2652
Test Run: Delete computer ID 2644
Getting computer with JSS ID 2644...

GENERAL INFORMATION:
Computer Name: SFO-M-PQ217G
Asset Number: None
JSS Computer ID: 2644
Serial Number:
Mac Address:
*
Managed: true
Traceback (most recent call last): File "/Volumes/CasperAPI/CasperAPI_CLI.py", line 1523, in <module> main() File "/Volumes/CasperAPI/CasperAPI_CLI.py", line 1439, in main deleteComputerIDsFromCSV(computersCSV, user, password) File "/Volumes/CasperAPI/CasperAPI_CLI.py", line 1206, in deleteComputerIDsFromCSV deleteComputerByID(compID, username, password) File "/Volumes/CasperAPI/CasperAPI_CLI.py", line 1166, in deleteComputerByID getComputerByID(comp_id, username, password) File "/Volumes/CasperAPI/CasperAPI_CLI.py", line 377, in getComputerByID print 'Last Check-In: ' + last_contact_time
TypeError: cannot concatenate 'str' and 'NoneType' objects

jkuo
Contributor

@Br3ck I'm glad it's helping you save at least a little time!

I'm considering making a way to take away the confirmation message (just wanted to be super safe there!) and perhaps displaying a column list of computer information and allowing you to click "Yes" to the whole list. Would that be helpful for you?

I'll fix that bug on the last check in - if the last_contact_time is null, then I need to process the display computer process differently.

Thanks,
Jason

Br3ck
New Contributor III

@jkuo - sorry for the ridiculous delay here, I appreciate the extra check before deleting but yeah i think a mechanism without that would be amazing seeing as how this is still an open PI with jamf.

CasperSally
Valued Contributor II

Really disappointed PI-004957 isn't fixed in 10.5. Is there a roadmap/time frame for getting this fixed?

mschroder
Valued Contributor

Deleting ~260 computers yesterday took about 12 hours, that is really really bad. And yes, the user and location history is completely out of control. An entry every few seconds, and the complete history since the big bang don't go well together.

CasperSally
Valued Contributor II

Is this PI planned to be fixed in 10.6? 10.7?

donmontalvo
Esteemed Contributor II

We just upgraded from 9.101.4 to 10.4.1 and we have a bunch of computers to delete...fingers crossed this got fixed.

--
https://donmontalvo.com

mbayhylle
New Contributor II

Week before last I was trying to delete computers, both individually and as a bulk action. That's when I learned about the above problem regarding bloated table sizes. I spent most of last week trying to re-import, repair following the steps provided by @rtrouton here: https://derflounder.wordpress.com/2014/02/01/fixing-caspers-mysql-database-with-mysqlcheck/, upgrading MySQL and macOS, and any number of other prayers.

During this process, I discovered that a couple of tables had become exceedingly bloated, particularly the jamfsoftware.applications table. That table itself ended up being 10.3GB and the mysqlcheck process ran for over 36 hours on a development machine before I gave up on it and followed the process used by @brunerd in this post: https://www.jamf.com/jamf-nation/discussions/17908/user-and-location-history-flush-maintenance

For my particular table I used the following queries:

###########
# flush reports, keep newest reports for each computer only
# count before
select count(*) from reports;
#make a new table with all the same columns
create table reports_new like reports;
#copy only the newest location_history for each computer
insert into reports_new select * from reports where report_id in (SELECT MAX(report_id) FROM reports GROUP BY computer_id);
#table switcheroo
rename table reports TO reports_old, reports_new TO reports;
# count after
select count(*) from reports;

#if all looks good
#drop table reports_old

###########
# flush applications, keep only newest report_id found in flushed reports
# count before
select count(*) from applications;
#make a new table with all the same columns
create table applications_new like applications;
#make new table from the remaining newest report_id
insert into applications_new select * from applications where report_id in (select report_id from reports);
#table switcheroo
rename table applications TO applications_old, applications_new TO applications;
#count after
select count(*) from applications;

#if all looks good
#drop table applications_old

After I did that the jamfsoftware.applications table returned to a normal size, I was able to run mysqlcheck -repair and mysqlcheck -o (optimize) on the database, and delete computers individually and using the bulk delete action.

So, long story short, because of the product issue in 10.5 you might want to review your current database to see if you have any tables that have grown to a ridiculous size, especially if you run into problems deleting computers.

emily
Valued Contributor III
Valued Contributor III

Spruce can also do cleanup of computers (and other things, like policies) fwiw.

tnielsen
Valued Contributor

JIIINGG YAAAANGGG!

scafide
Contributor III
Contributor III

This is resolved in Jamf Pro 10.6 as referenced in the Beta Release Notes.

donmontalvo
Esteemed Contributor II

@scafide thanks for confirming...can't wait for 10.6.1. ;)

--
https://donmontalvo.com

rtrouton
Valued Contributor III

I have a post on how I'm doing mass-deletion via the API:

https://derflounder.wordpress.com/2018/05/19/using-the-jamf-pro-api-to-mass-delete-computers-and-mob...

That said, I'm very happy to hear this is fixed in 10.6.x.

donmontalvo
Esteemed Contributor II

@rtrouton thanks, Jamf Support told us Product Issue PI-004957 impacts both GUI and API mass deletion.

--
https://donmontalvo.com

rtrouton
Valued Contributor III

Sure, which is why my solution is deleting one ID at a time, which doesn't run afoul of PI-004957.

tnielsen
Valued Contributor

Hey guys, mine is working but I had to wait about 20minutes for it to process.

donmontalvo
Esteemed Contributor II

@rtrouton ah...I should have RTFB (Read The Fine Blog). :) My apologies, I assumed wrongly. Will test your script today. Thanks!

--
https://donmontalvo.com

donmontalvo
Esteemed Contributor II

We ended up taking parts of @rtrouton's script and some other scripts found here on Jamf Nation and tweaking for our needs.

Basically save a report with the computers you want to delete, and with only JSS ID checked, then put it in a folder with our script, and let it rip. After script finishes, run the report again to confirm it shows zero computers.

DISCLAIMER: We hard code the URL to avoid any mistakes, this way we can give a walk through pointing to our QA before running in Production.

#!/bin/bash

currentDir=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
reportFile="$1"
tempFile="/tmp/tempFile.csv"
inputFile="${currentDir}/inputFile.csv"
jamfproURL="https://server.domain.com:8443"
jamfproIDURL="${jamfproURL}/JSSResource/computers/id"
jamfproURL=${jamfproURL%%/}

# cd to working folder
cd "$currentDir"

# Remove header
sed '1d' "${reportFile}" > ${tempFile}

# Trim list to just JSS ID numbers
cat ${tempFile} | cut -f2 -d"," > ${inputFile}

# Ensure csv has a line return at end of file
if [[ -n "$(tail -c 1 "${inputFile}")" ]]
then
    sed -i '' -e "'$a'" "${inputFile}"
fi

# Prompt for credentials
if [[ -n ${inputFile} && -r ${inputFile} ]]; then

    echo ""
    echo "Enter username:"
    read apiUser

    echo ""
    echo "Enter password:"
    read apiPass
    echo ""

    # Loop through list to remove JSS IDs from JSS
    while read -r ID
    do
        if [[ "$ID" =~ ^[0-9]+$ ]]; then
            curl -X DELETE "${jamfproIDURL}/$ID" -u ${apiUser}:${apiPass}
        else
            echo "Sorry every line in file needs to be a number!"
        fi
    done < ${inputFile}
else
    # Complain if the list is missing or wrong format.
    echo "Input file is missing or not the right format. Contact Mac Engineering."
    ERROR=1
fi

exit 0

At some point I plan to try to clean up (or simplify) the output of success and error.

Currently here is what you get if you have a list of 3 computers that exist:

<?xml version="1.0" encoding="UTF-8"?><computer><id>10100</id></computer><?xml version="1.0" encoding="UTF-8"?><computer><id>10101</id></computer><?xml version="1.0" encoding="UTF-8"?><computer><id>10102</id></computer>

And here is what you get if you have a list of 3 computers that do not exist:

<html>
<head>
   <title>Status page</title>
</head>
<body style="font-family: sans-serif;">
<p style="font-size: 1.2em;font-weight: bold;margin: 1em 0px;">Not Found</p>
<p>The server has not found anything matching the request URI</p>
<p>You can get technical details <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5">here</a>.<br>
Please continue your visit at our <a href="/">home page</a>.
</p>
</body>
</html>
<html>
<head>
   <title>Status page</title>
</head>
<body style="font-family: sans-serif;">
<p style="font-size: 1.2em;font-weight: bold;margin: 1em 0px;">Not Found</p>
<p>The server has not found anything matching the request URI</p>
<p>You can get technical details <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5">here</a>.<br>
Please continue your visit at our <a href="/">home page</a>.
</p>
</body>
</html>
<html>
<head>
   <title>Status page</title>
</head>
<body style="font-family: sans-serif;">
<p style="font-size: 1.2em;font-weight: bold;margin: 1em 0px;">Not Found</p>
<p>The server has not found anything matching the request URI</p>
<p>You can get technical details <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.5">here</a>.<br>
Please continue your visit at our <a href="/">home page</a>.
</p>
</body>
</html>

HTH,
Don

--
https://donmontalvo.com

Cem
Valued Contributor

thanks for the script :)